Jul 21 17:25:46.808856 ip-10-0-135-228 systemd[1]: Starting Kubernetes Kubelet...
Jul 21 17:25:47.187319 ip-10-0-135-228 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 21 17:25:47.187319 ip-10-0-135-228 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 21 17:25:47.187319 ip-10-0-135-228 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 21 17:25:47.187319 ip-10-0-135-228 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 21 17:25:47.187319 ip-10-0-135-228 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 21 17:25:47.188376 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.188253    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 21 17:25:47.193384 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193362    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:25:47.193384 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193381    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:25:47.193384 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193386    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:25:47.193384 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193391    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193395    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193399    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193404    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193407    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193411    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193414    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193418    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193424    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193427    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193431    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193435    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193439    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193443    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193446    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193450    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193454    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193457    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193461    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193465    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:25:47.193659 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193469    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193473    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193477    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193481    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193485    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193489    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193492    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193497    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193503    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193509    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193514    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193518    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193522    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193527    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193532    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193552    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193557    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193561    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193567    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:25:47.194453 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193572    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193580    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193586    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193591    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193596    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193600    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193605    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193609    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193613    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193617    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193621    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193626    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193630    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193635    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193639    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193643    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193648    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193652    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193656    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193661    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:25:47.194955 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193665    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193669    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193674    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193678    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193682    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193686    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193692    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193698    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193702    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193709    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193715    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193719    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193724    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193730    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193734    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193738    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193743    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193747    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193751    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:25:47.195646 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193756    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193760    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193764    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193768    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193772    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193777    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193781    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193786    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193790    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193794    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193798    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193802    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193808    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193812    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193817    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193821    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193825    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.193829    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195862    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:25:47.196491 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195874    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195880    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195885    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195891    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195895    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195903    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195908    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195914    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195919    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195924    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195928    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195933    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195937    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195942    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195946    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195951    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195955    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195959    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195963    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:25:47.197290 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195967    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195972    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195976    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195980    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195984    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195989    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195994    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.195998    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196002    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196006    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196010    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196015    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196019    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196023    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196027    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196031    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196035    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196039    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196043    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196047    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:25:47.198111 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196053    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196057    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196061    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196065    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196069    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196073    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196078    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196082    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196091    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196095    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196099    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196103    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196107    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196112    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196116    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196121    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196125    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196129    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196134    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196138    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:25:47.198991 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196142    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196147    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196151    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196155    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196159    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196163    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196167    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196170    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196175    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196179    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196183    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196187    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196191    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196199    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196204    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196211    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196215    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196219    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196224    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:25:47.199578 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196229    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196235    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196239    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196244    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196248    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196252    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196256    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196260    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196264    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196269    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196273    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196277    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196281    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196285    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196289    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196293    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196297    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196301    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196305    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196312    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:25:47.200228 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196985    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196993    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.196998    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197002    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197007    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197011    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197018    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197024    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197030    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197035    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197040    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197044    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197048    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197055    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197059    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197063    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197067    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197072    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197076    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:25:47.200861 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197080    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197084    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197089    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197093    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197098    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197102    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197106    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197110    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197115    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197121    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197126    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197131    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197136    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197141    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197145    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197149    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197153    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197157    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197161    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197165    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:25:47.201315 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197169    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197174    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197179    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197183    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197188    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197192    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197198    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197203    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197207    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197211    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197215    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197219    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197224    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197228    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197232    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197236    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197241    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197245    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197249    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197252    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:25:47.201917 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197256    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197261    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197265    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197270    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197274    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197278    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197282    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197286    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197290    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197294    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197298    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197302    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197305    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197309    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197313    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197318    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197322    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197327    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197331    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197336    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:25:47.202432 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197340    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197343    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197347    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197351    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197357    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197361    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197365    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197369    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197373    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197377    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197386    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197391    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197394    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197400    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197406    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197411    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197416    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197420    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197424    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:25:47.202949 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.197428    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197552    2562 flags.go:64] FLAG: --address="0.0.0.0"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197564    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197574    2562 flags.go:64] FLAG: --anonymous-auth="true"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197580    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197587    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197593    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197600    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197607    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197614    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197619    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197625    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197630    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197635    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197640    2562 flags.go:64] FLAG: --cgroup-root=""
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197644    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197649    2562 flags.go:64] FLAG: --client-ca-file=""
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197653    2562 flags.go:64] FLAG: --cloud-provider="external"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197658    2562 flags.go:64] FLAG: --cluster-dns="[]"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197665    2562 flags.go:64] FLAG: --cluster-domain=""
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197670    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197675    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197680    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197685    2562 flags.go:64] FLAG: --container-log-max-files="5"
Jul 21 17:25:47.203389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197691    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197696    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197702    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197707    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197712    2562 flags.go:64] FLAG: --contention-profiling="false"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197717    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197721    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197726    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197731    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197738    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197743    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197747    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197752    2562 flags.go:64] FLAG: --enable-load-reader="false"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197757    2562 flags.go:64] FLAG: --enable-server="true"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197761    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197770    2562 flags.go:64] FLAG: --event-burst="100"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197775    2562 flags.go:64] FLAG: --event-qps="50"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197780    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197785    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197790    2562 flags.go:64] FLAG: --eviction-hard=""
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197796    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197801    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197806    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197811    2562 flags.go:64] FLAG: --eviction-soft=""
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197816    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 21 17:25:47.203979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197821    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197825    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197830    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197834    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197839    2562 flags.go:64] FLAG: --fail-swap-on="true"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197844    2562 flags.go:64] FLAG: --feature-gates=""
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197850    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197856    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197861    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197866    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197870    2562 flags.go:64] FLAG: --healthz-port="10248"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197875    2562 flags.go:64] FLAG: --help="false"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197880    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197885    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197890    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197895    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197900    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197905    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197910    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197914    2562 flags.go:64] FLAG: --image-service-endpoint=""
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197918    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197923    2562 flags.go:64] FLAG: --kube-api-burst="100"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197931    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197936    2562 flags.go:64] FLAG: --kube-api-qps="50"
Jul 21 17:25:47.204639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197941    2562 flags.go:64] FLAG: --kube-reserved=""
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197946    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197951    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197956    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197961    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197966    2562 flags.go:64] FLAG: --lock-file=""
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197970    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197975    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197980    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197989    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197993    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.197998    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198003    2562 flags.go:64] FLAG: --logging-format="text"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198007    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198013    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198018    2562 flags.go:64] FLAG: --manifest-url=""
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198022    2562 flags.go:64] FLAG: --manifest-url-header=""
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198029    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198033    2562 flags.go:64] FLAG: --max-open-files="1000000"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198040    2562 flags.go:64] FLAG: --max-pods="110"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198045    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198050    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198054    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198059    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198063    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 21 17:25:47.205294 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198068    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198072    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198085    2562 flags.go:64] FLAG: --node-status-max-images="50"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198090    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198094    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198101    2562 flags.go:64] FLAG: --pod-cidr=""
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198105    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198114    2562 flags.go:64] FLAG: --pod-manifest-path=""
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198118    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198124    2562 flags.go:64] FLAG: --pods-per-core="0"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198129    2562 flags.go:64] FLAG: --port="10250"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198134    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198139    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cf989bfa2c3e1c66"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198144    2562 flags.go:64] FLAG: --qos-reserved=""
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198149    2562 flags.go:64] FLAG: --read-only-port="10255"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198154    2562 flags.go:64] FLAG: --register-node="true"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198159    2562 flags.go:64] FLAG: --register-with-taints=""
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198165    2562 flags.go:64] FLAG: --registry-burst="10"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198169    2562 flags.go:64] FLAG: --registry-qps="5"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198175    2562 flags.go:64] FLAG: --reserved-cpus=""
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198179    2562 flags.go:64] FLAG: --reserved-memory=""
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198186    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198192    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198197    2562 flags.go:64] FLAG: --rotate-certificates="false"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198202    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 21 17:25:47.205916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198206    2562 flags.go:64] FLAG: --runonce="false"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198211    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198216    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198221    2562 flags.go:64] FLAG: --seccomp-default="false"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198225    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198229    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198234    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198239    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198244    2562 flags.go:64] FLAG: --storage-driver-password="root"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198248    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198253    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198257    2562 flags.go:64] FLAG: --storage-driver-user="root"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198265    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198271    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198276    2562 flags.go:64] FLAG: --system-cgroups=""
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198280    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198290    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198295    2562 flags.go:64] FLAG: --tls-cert-file=""
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198301    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198307    2562 flags.go:64] FLAG: --tls-min-version=""
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198311    2562 flags.go:64] FLAG: --tls-private-key-file=""
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198316    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198321    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198325    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198330    2562 flags.go:64] FLAG: --v="2"
Jul 21 17:25:47.206587 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198336    2562 flags.go:64] FLAG: --version="false"
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198342    2562 flags.go:64] FLAG: --vmodule=""
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198348    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.198353    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198558    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198565    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198570    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198574    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198578    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198583    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198587    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198592    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198596    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198603    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198609    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198613    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198617    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198622    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198627    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198633    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:25:47.207181 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198638    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198642    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198647    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198655    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198660    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198665    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198672    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198677    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198682    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198686    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198690    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198694    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198699    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198703    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198707    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198711    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198715    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198719    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198723    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:25:47.207671 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198727    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198731    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198735    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198740    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198744    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198748    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198752    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198756    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198760    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198764    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198768    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198772    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198777    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198782    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198786    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198790    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198795    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198800    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198804    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:25:47.208115 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198810    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198817    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198822    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198827    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198831    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198835    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198839    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198843    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198847    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198851    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198855    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198859    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198863    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198868    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198872    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198876    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198880    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198884    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198888    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198892    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:25:47.208596 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198896    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198899    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198903    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198914    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198918    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198924    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198928    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198932    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198936    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198942    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198946    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198950    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198954    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198958    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198964    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198968    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198972    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198977    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198981    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198985    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:25:47.209070 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198989    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:25:47.209563 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198992    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:25:47.209563 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.198996    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:25:47.209563 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.199000    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:25:47.209563 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.199003    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:25:47.209563 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.199016    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 21 17:25:47.209563 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.206395    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 21 17:25:47.209563 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.206415    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 21 17:25:47.209563 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.207309    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 21 17:25:47.209563 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.207316    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 21 17:25:47.209563 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207377    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:25:47.209563 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207382    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:25:47.209563 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207385    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:25:47.209563 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207389    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:25:47.209563 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207392    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:25:47.209563 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207395    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207398    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207401    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207404    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207407    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207410    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207413    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207416    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207418    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207421    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207424    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207426    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207429    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207431    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207434    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207437    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207439    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207442    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207444    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207447    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:25:47.209931 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207449    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207452    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207454    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207457    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207461    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207464    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207466    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207469    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207473    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207477    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207480    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207482    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207485    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207487    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207490    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207492    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207495    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207499    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207502    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:25:47.210402 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207505    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207507    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207510    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207513    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207516    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207518    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207521    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207523    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207525    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207528    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207530    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207533    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207547    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207550    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207552    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207555    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207559    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207563    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207566    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:25:47.210950 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207569    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207572    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207575    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207577    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207580    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207582    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207585    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207588    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207590    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207592    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207595    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207597    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207599    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207602    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207604    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207607    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207609    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207612    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207614    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207617    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:25:47.211406 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207619    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:25:47.211906 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207622    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:25:47.211906 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207624    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:25:47.211906 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207626    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:25:47.211906 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207629    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:25:47.211906 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207631    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:25:47.211906 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207634    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:25:47.211906 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207636    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:25:47.211906 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207639    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:25:47.211906 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207641    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:25:47.211906 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207645    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:25:47.211906 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207647    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:25:47.211906 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207650    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:25:47.211906 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207653    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:25:47.211906 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207655    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:25:47.211906 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207658    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:25:47.212261 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.207663    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 21 17:25:47.212261 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207779    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:25:47.212261 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207784    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:25:47.212261 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207787    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:25:47.212261 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207789    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:25:47.212261 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207792    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:25:47.212261 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207795    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:25:47.212261 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207797    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:25:47.212261 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207800    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:25:47.212261 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207802    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:25:47.212261 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207804    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:25:47.212261 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207807    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:25:47.212261 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207810    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:25:47.212261 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207812    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:25:47.212261 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207815    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207817    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207820    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207822    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207824    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207827    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207829    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207832    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207834    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207837    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207839    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207842    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207844    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207847    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207850    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207853    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207856    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207858    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207861    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207863    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:25:47.212635 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207866    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207868    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207871    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207873    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207876    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207878    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207881    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207883    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207885    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207888    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207891    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207893    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207896    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207898    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207900    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207903    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207907    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207911    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207913    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:25:47.213118 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207916    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207919    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207922    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207924    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207927    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207929    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207932    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207935    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207939    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207944    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207947    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207949    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207951    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207954    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207956    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207959    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207961    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207965    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207968    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:25:47.213584 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207971    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207974    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207976    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207979    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207982    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207984    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207987    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207989    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207992    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207994    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207996    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.207999    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.208001    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.208004    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.208006    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.208008    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.208011    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.208014    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.208016    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:25:47.214026 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.208019    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:25:47.214718 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.208022    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:25:47.214718 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.208024    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:25:47.214718 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.208027    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:25:47.214718 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.208029    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:25:47.214718 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.208032    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:25:47.214718 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.208034    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:25:47.214718 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.208037    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:25:47.214718 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:47.208039    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:25:47.214718 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.208044    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 21 17:25:47.214718 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.208160    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 21 17:25:47.214718 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.211509    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 21 17:25:47.214718 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.212677    2562 server.go:1019] "Starting client certificate rotation"
Jul 21 17:25:47.214718 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.212777    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 21 17:25:47.214718 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.212812    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 21 17:25:47.217260 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.217240    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 21 17:25:47.222329 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.222311    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 21 17:25:47.234458 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.234439    2562 log.go:25] "Validated CRI v1 runtime API"
Jul 21 17:25:47.236666 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.236651    2562 log.go:25] "Validated CRI v1 image API"
Jul 21 17:25:47.237475 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.237461    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 21 17:25:47.239885 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.239865    2562 fs.go:135] Filesystem UUIDs: map[6d6e3588-f1e0-4330-b594-4790027502fa:/dev/nvme0n1p4 76c59f4e-5c4a-4b85-ae21-270bb756726d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 21 17:25:47.239944 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.239884    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 21 17:25:47.245084 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.245067    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 21 17:25:47.245954 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.245847    2562 manager.go:216] Machine: {Timestamp:2026-07-21 17:25:47.244568119 +0000 UTC m=+0.325715238 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100406 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec249a95c4a575ec851cac952b8e9474 SystemUUID:ec249a95-c4a5-75ec-851c-ac952b8e9474 BootID:d465a495-555c-4f79-bf25-753df7b99236 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:26:1a:3d:b6:27 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:26:1a:3d:b6:27 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:db:be:fe:02:d6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 21 17:25:47.245954 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.245953    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 21 17:25:47.246040 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.246027    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 21 17:25:47.248834 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.248805    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 21 17:25:47.248978 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.248839    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-228.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 21 17:25:47.249960 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.249950    2562 topology_manager.go:138] "Creating topology manager with none policy"
Jul 21 17:25:47.249992 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.249962    2562 container_manager_linux.go:309] "Creating device plugin manager"
Jul 21 17:25:47.249992 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.249970    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 21 17:25:47.250616 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.250605    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 21 17:25:47.251671 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.251661    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 21 17:25:47.254003 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.253992    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 21 17:25:47.254117 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.254108    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 21 17:25:47.254147 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.254137    2562 kubelet.go:486] "Attempting to sync node with API server"
Jul 21 17:25:47.254174 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.254157    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 21 17:25:47.254174 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.254168    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 21 17:25:47.254232 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.254191    2562 kubelet.go:398] "Adding apiserver pod source"
Jul 21 17:25:47.254232 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.254199    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 21 17:25:47.254619 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.254607    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 21 17:25:47.254665 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.254627    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 21 17:25:47.256728 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.256713    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 21 17:25:47.259406 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.259378    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 21 17:25:47.259492 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.259437    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 21 17:25:47.260403 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.260386    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 21 17:25:47.260500 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.260410    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 21 17:25:47.260500 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.260421    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 21 17:25:47.260500 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.260431    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 21 17:25:47.260500 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.260459    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 21 17:25:47.260500 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.260474    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 21 17:25:47.260500 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.260486    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 21 17:25:47.260500 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.260492    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 21 17:25:47.260500 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.260500    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 21 17:25:47.260500 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.260506    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 21 17:25:47.260772 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.260515    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 21 17:25:47.260772 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.260521    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 21 17:25:47.261677 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.261658    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 21 17:25:47.261750 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.261691    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 21 17:25:47.264928 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.264912    2562 server.go:1268] "Started kubelet"
Jul 21 17:25:47.265184 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.265136    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 21 17:25:47.265278 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.265112    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 21 17:25:47.265278 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.265268    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 21 17:25:47.265891 ip-10-0-135-228 systemd[1]: Started Kubernetes Kubelet.
Jul 21 17:25:47.267554 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.267518    2562 server.go:310] "Adding debug handlers to kubelet server"
Jul 21 17:25:47.268199 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.268167    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 21 17:25:47.268293 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.268256    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-228.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 21 17:25:47.268367 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.268319    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 21 17:25:47.269524 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.269500    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 21 17:25:47.269817 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.269795    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 21 17:25:47.270675 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.270660    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jw9ft"
Jul 21 17:25:47.271099 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.270256    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-228.ec2.internal.18c45e0981702023  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-228.ec2.internal,UID:ip-10-0-135-228.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-228.ec2.internal,},FirstTimestamp:2026-07-21 17:25:47.264884771 +0000 UTC m=+0.346031875,LastTimestamp:2026-07-21 17:25:47.264884771 +0000 UTC m=+0.346031875,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-228.ec2.internal,}"
Jul 21 17:25:47.274381 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.274369    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 21 17:25:47.274429 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.274394    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 21 17:25:47.274513 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.274493    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-228.ec2.internal\" not found"
Jul 21 17:25:47.274890 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.274872    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 21 17:25:47.274890 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.274885    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 21 17:25:47.274890 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.274891    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 21 17:25:47.275069 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.275017    2562 reconstruct.go:97] "Volume reconstruction finished"
Jul 21 17:25:47.275069 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.275025    2562 reconciler.go:29] "Reconciler: start to sync state"
Jul 21 17:25:47.275355 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.275338    2562 factory.go:55] Registering systemd factory
Jul 21 17:25:47.275433 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.275359    2562 factory.go:223] Registration of the systemd container factory successfully
Jul 21 17:25:47.275581 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.275568    2562 factory.go:153] Registering CRI-O factory
Jul 21 17:25:47.275581 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.275582    2562 factory.go:223] Registration of the crio container factory successfully
Jul 21 17:25:47.275699 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.275624    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 21 17:25:47.275699 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.275652    2562 factory.go:103] Registering Raw factory
Jul 21 17:25:47.275699 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.275667    2562 manager.go:1196] Started watching for new ooms in manager
Jul 21 17:25:47.276150 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.276134    2562 manager.go:349] Starting recovery of all containers
Jul 21 17:25:47.279460 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.279313    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jw9ft"
Jul 21 17:25:47.279565 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.279458    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 21 17:25:47.279633 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.279572    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 21 17:25:47.285168 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.285144    2562 manager.go:354] Recovery completed
Jul 21 17:25:47.291082 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.291068    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:25:47.293973 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.293956    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:25:47.294038 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.293990    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:25:47.294038 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.294004    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:25:47.294509 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.294494    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 21 17:25:47.294509 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.294508    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 21 17:25:47.294649 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.294523    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 21 17:25:47.294649 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.294519    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:25:47.295647 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.295636    2562 policy_none.go:49] "None policy: Start"
Jul 21 17:25:47.295688 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.295652    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 21 17:25:47.295688 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.295663    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 21 17:25:47.296225 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.296152    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-228.ec2.internal.18c45e09832bfd7e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-228.ec2.internal,UID:ip-10-0-135-228.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-228.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-228.ec2.internal,},FirstTimestamp:2026-07-21 17:25:47.293973886 +0000 UTC m=+0.375120991,LastTimestamp:2026-07-21 17:25:47.293973886 +0000 UTC m=+0.375120991,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-228.ec2.internal,}"
Jul 21 17:25:47.296305 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.296242    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:25:47.296305 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.296265    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:25:47.296305 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.296275    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:25:47.296656 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.296643    2562 policy_none.go:47] "Start"
Jul 21 17:25:47.332248 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.332082    2562 manager.go:337] "Starting Device Plugin manager"
Jul 21 17:25:47.332353 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.332276    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 21 17:25:47.332353 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.332291    2562 server.go:87] "Starting device plugin registration server"
Jul 21 17:25:47.332572 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.332534    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 21 17:25:47.332656 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.332576    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 21 17:25:47.332715 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.332674    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 21 17:25:47.332762 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.332756    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 21 17:25:47.332807 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.332765    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 21 17:25:47.333231 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.333213    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 21 17:25:47.333313 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.333266    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-228.ec2.internal\" not found"
Jul 21 17:25:47.404596 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.404555    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 21 17:25:47.405866 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.405841    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 21 17:25:47.405973 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.405894    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 21 17:25:47.405973 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.405929    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 21 17:25:47.406084 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.405972    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 21 17:25:47.408682 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.408661    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:25:47.432938 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.432913    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:25:47.433781 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.433766    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:25:47.433856 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.433798    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:25:47.433856 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.433811    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:25:47.433856 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.433836    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.443185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.443141    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.443185 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.443167    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-228.ec2.internal\": node \"ip-10-0-135-228.ec2.internal\" not found"
Jul 21 17:25:47.460959 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.460934    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-228.ec2.internal\" not found"
Jul 21 17:25:47.507000 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.506975    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-228.ec2.internal"]
Jul 21 17:25:47.507080 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.507036    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:25:47.507880 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.507866    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:25:47.507929 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.507897    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:25:47.507929 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.507913    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:25:47.509032 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.509020    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:25:47.509152 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.509140    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.509196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.509167    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:25:47.509690 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.509678    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:25:47.509749 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.509701    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:25:47.509749 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.509710    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:25:47.510140 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.510122    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:25:47.510199 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.510150    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:25:47.510199 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.510161    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:25:47.511093 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.511080    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.511168 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.511102    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:25:47.511732 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.511714    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:25:47.511817 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.511746    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:25:47.511817 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.511758    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:25:47.541157 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.541129    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-228.ec2.internal\" not found" node="ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.545486 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.545469    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-228.ec2.internal\" not found" node="ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.561521 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.561502    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-228.ec2.internal\" not found"
Jul 21 17:25:47.576639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.576620    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91cb1fde35779c2f274066a1ceeb532c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal\" (UID: \"91cb1fde35779c2f274066a1ceeb532c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.576691 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.576651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91cb1fde35779c2f274066a1ceeb532c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal\" (UID: \"91cb1fde35779c2f274066a1ceeb532c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.576691 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.576673    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5c1883a5f27b82c72d9daa1f8b8a95db-config\") pod \"kube-apiserver-proxy-ip-10-0-135-228.ec2.internal\" (UID: \"5c1883a5f27b82c72d9daa1f8b8a95db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.661806 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.661774    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-228.ec2.internal\" not found"
Jul 21 17:25:47.677200 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.677167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91cb1fde35779c2f274066a1ceeb532c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal\" (UID: \"91cb1fde35779c2f274066a1ceeb532c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.677299 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.677223    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91cb1fde35779c2f274066a1ceeb532c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal\" (UID: \"91cb1fde35779c2f274066a1ceeb532c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.677299 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.677248    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5c1883a5f27b82c72d9daa1f8b8a95db-config\") pod \"kube-apiserver-proxy-ip-10-0-135-228.ec2.internal\" (UID: \"5c1883a5f27b82c72d9daa1f8b8a95db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.677299 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.677170    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91cb1fde35779c2f274066a1ceeb532c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal\" (UID: \"91cb1fde35779c2f274066a1ceeb532c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.677299 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.677296    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5c1883a5f27b82c72d9daa1f8b8a95db-config\") pod \"kube-apiserver-proxy-ip-10-0-135-228.ec2.internal\" (UID: \"5c1883a5f27b82c72d9daa1f8b8a95db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.677433 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.677303    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91cb1fde35779c2f274066a1ceeb532c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal\" (UID: \"91cb1fde35779c2f274066a1ceeb532c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.762674 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.762603    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-228.ec2.internal\" not found"
Jul 21 17:25:47.844172 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.844145    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.848591 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:47.847706    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-228.ec2.internal"
Jul 21 17:25:47.863008 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.862982    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-228.ec2.internal\" not found"
Jul 21 17:25:47.963573 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:47.963509    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-228.ec2.internal\" not found"
Jul 21 17:25:48.064121 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:48.064051    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-228.ec2.internal\" not found"
Jul 21 17:25:48.164601 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:48.164567    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-228.ec2.internal\" not found"
Jul 21 17:25:48.212110 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:48.212081    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 21 17:25:48.212522 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:48.212238    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 21 17:25:49.775035 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:49.775007    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-228.ec2.internal\" not found"
Jul 21 17:25:49.792219 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:49.792186    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 17:20:48 +0000 UTC" deadline="2028-03-16 05:46:22.943715866 +0000 UTC"
Jul 21 17:25:49.792219 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:49.792215    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14484h20m33.151504485s"
Jul 21 17:25:49.819208 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:49.819179    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:25:49.868503 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:49.868470    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1883a5f27b82c72d9daa1f8b8a95db.slice/crio-826583c36272f591534f61017ccd40575e9cdc94738e89bdef6c1243314d349c WatchSource:0}: Error finding container 826583c36272f591534f61017ccd40575e9cdc94738e89bdef6c1243314d349c: Status 404 returned error can't find the container with id 826583c36272f591534f61017ccd40575e9cdc94738e89bdef6c1243314d349c
Jul 21 17:25:49.868680 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:49.868659    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cb1fde35779c2f274066a1ceeb532c.slice/crio-fb091543f2412de3a8f3844f2a4beaf9f863b69764be27ca50bfcdb4d129292e WatchSource:0}: Error finding container fb091543f2412de3a8f3844f2a4beaf9f863b69764be27ca50bfcdb4d129292e: Status 404 returned error can't find the container with id fb091543f2412de3a8f3844f2a4beaf9f863b69764be27ca50bfcdb4d129292e
Jul 21 17:25:49.872963 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:49.872945    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 17:25:49.875600 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:49.875585    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-228.ec2.internal\" not found"
Jul 21 17:25:49.918958 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:49.918904    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-228.ec2.internal" event={"ID":"5c1883a5f27b82c72d9daa1f8b8a95db","Type":"ContainerStarted","Data":"826583c36272f591534f61017ccd40575e9cdc94738e89bdef6c1243314d349c"}
Jul 21 17:25:49.919960 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:49.919934    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal" event={"ID":"91cb1fde35779c2f274066a1ceeb532c","Type":"ContainerStarted","Data":"fb091543f2412de3a8f3844f2a4beaf9f863b69764be27ca50bfcdb4d129292e"}
Jul 21 17:25:49.976154 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:49.976114    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-228.ec2.internal\" not found"
Jul 21 17:25:49.983436 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:49.983416    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:25:49.984352 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:49.984334    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal"
Jul 21 17:25:49.995136 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:49.995112    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 21 17:25:49.996139 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:49.996127    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-228.ec2.internal"
Jul 21 17:25:50.003271 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.003255    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 21 17:25:50.323927 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.323893    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:25:50.765325 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.765290    2562 apiserver.go:52] "Watching apiserver"
Jul 21 17:25:50.778614 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.778589    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 21 17:25:50.779008 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.778978    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-zl482","kube-system/kube-apiserver-proxy-ip-10-0-135-228.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt","openshift-cluster-node-tuning-operator/tuned-rkbgr","openshift-dns/node-resolver-8gvpd","openshift-image-registry/node-ca-xxbb8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal","openshift-multus/multus-additional-cni-plugins-7rnxt","kube-system/global-pull-secret-syncer-qht4v","openshift-multus/multus-tx58r","openshift-multus/network-metrics-daemon-9r9z4","openshift-network-diagnostics/network-check-target-r6qp2","openshift-network-operator/iptables-alerter-5gtz4","openshift-ovn-kubernetes/ovnkube-node-bdplm"]
Jul 21 17:25:50.781298 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.781278    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xxbb8"
Jul 21 17:25:50.783846 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.783828    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:25:50.783936 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:50.783900    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:25:50.785192 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.785107    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 21 17:25:50.785914 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.785775    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 21 17:25:50.785914 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.785790    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 21 17:25:50.785914 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.785781    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xpfvp\""
Jul 21 17:25:50.786058 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.785943    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 21 17:25:50.786893 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.786834    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.789125 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.789106    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:50.789465 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.789452    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 21 17:25:50.789540 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.789484    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 21 17:25:50.789600 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.789567    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k8plg\""
Jul 21 17:25:50.789650 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.789622    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 21 17:25:50.791725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.791707    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 21 17:25:50.791837 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.791816    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kxl5w\""
Jul 21 17:25:50.791976 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.791846    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 21 17:25:50.792157 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.792138    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:25:50.792236 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:50.792210    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:25:50.794489 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.794468    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zl482"
Jul 21 17:25:50.794739 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.794472    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 21 17:25:50.795124 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.795101    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-socket-dir\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.795197 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.795138    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svxwl\" (UniqueName: \"kubernetes.io/projected/bbf98769-a9a3-4a65-8a28-33809628690b-kube-api-access-svxwl\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.795197 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.795165    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b35c0345-124e-40c5-8de6-2b9c3122e24a-serviceca\") pod \"node-ca-xxbb8\" (UID: \"b35c0345-124e-40c5-8de6-2b9c3122e24a\") " pod="openshift-image-registry/node-ca-xxbb8"
Jul 21 17:25:50.795197 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.795186    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-registration-dir\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.795309 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.795208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-device-dir\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.795309 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.795228    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-etc-selinux\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.795309 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.795242    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-sys-fs\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.795309 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.795262    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b35c0345-124e-40c5-8de6-2b9c3122e24a-host\") pod \"node-ca-xxbb8\" (UID: \"b35c0345-124e-40c5-8de6-2b9c3122e24a\") " pod="openshift-image-registry/node-ca-xxbb8"
Jul 21 17:25:50.795309 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.795287    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr6gp\" (UniqueName: \"kubernetes.io/projected/b35c0345-124e-40c5-8de6-2b9c3122e24a-kube-api-access-zr6gp\") pod \"node-ca-xxbb8\" (UID: \"b35c0345-124e-40c5-8de6-2b9c3122e24a\") " pod="openshift-image-registry/node-ca-xxbb8"
Jul 21 17:25:50.795502 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.795319    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wndt4\" (UniqueName: \"kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4\") pod \"network-check-target-r6qp2\" (UID: \"c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4\") " pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:25:50.795502 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.795343    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.797349 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.796762    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:50.797349 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.797019    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 21 17:25:50.797349 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.797037    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8vqt7\""
Jul 21 17:25:50.797349 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.797171    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 21 17:25:50.798986 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.798969    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:50.799059 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:50.799030    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:25:50.800029 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.799796    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 21 17:25:50.800029 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.799833    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 21 17:25:50.800029 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.799865    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 21 17:25:50.800029 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.799904    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-z7jvt\""
Jul 21 17:25:50.800278 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.800109    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 21 17:25:50.800278 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.800135    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 21 17:25:50.801581 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.801563    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.803672 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.803654    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8gvpd"
Jul 21 17:25:50.804159 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.804131    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ng26v\""
Jul 21 17:25:50.804249 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.804131    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 21 17:25:50.805904 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.805884    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-655g9\""
Jul 21 17:25:50.806141 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.806120    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 21 17:25:50.806231 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.806149    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 21 17:25:50.806396 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.806376    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5gtz4"
Jul 21 17:25:50.808592 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.808574    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 21 17:25:50.808749 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.808728    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.808911 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.808892    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6p9l8\""
Jul 21 17:25:50.809028 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.809015    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 21 17:25:50.809309 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.809291    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 21 17:25:50.811474 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.811456    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f22ww\""
Jul 21 17:25:50.811697 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.811678    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 21 17:25:50.811999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.811978    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 21 17:25:50.812111 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.811998    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 21 17:25:50.812220 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.812206    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 21 17:25:50.812279 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.812255    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 21 17:25:50.812399 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.812382    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 21 17:25:50.816806 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.816784    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wbtmx"
Jul 21 17:25:50.824692 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.824672    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wbtmx"
Jul 21 17:25:50.885427 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.885402    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 21 17:25:50.895806 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.895598    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-modprobe-d\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:50.895806 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.895637    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-log-socket\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.895806 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.895664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk62t\" (UniqueName: \"kubernetes.io/projected/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-kube-api-access-tk62t\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.895806 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.895689    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-os-release\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.895806 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.895707    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e623c85-720f-4764-9898-3a0ecc93a45a-cni-binary-copy\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.895806 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.895722    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-run-netns\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.895806 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.895743    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-run-multus-certs\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.895806 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.895793    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-tuned\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:50.896304 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.895826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs\") pod \"network-metrics-daemon-9r9z4\" (UID: \"cb83c0b0-6020-43bf-be65-b41d51b76e5a\") " pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:25:50.896304 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.895854    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-os-release\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:50.896304 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.895879    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-systemd-units\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.896304 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.895905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-544bm\" (UniqueName: \"kubernetes.io/projected/85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15-kube-api-access-544bm\") pod \"node-resolver-8gvpd\" (UID: \"85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15\") " pod="openshift-dns/node-resolver-8gvpd"
Jul 21 17:25:50.896304 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.895928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7lmb\" (UniqueName: \"kubernetes.io/projected/a016f87b-eabf-47b0-a16f-94dafd80e108-kube-api-access-l7lmb\") pod \"iptables-alerter-5gtz4\" (UID: \"a016f87b-eabf-47b0-a16f-94dafd80e108\") " pod="openshift-network-operator/iptables-alerter-5gtz4"
Jul 21 17:25:50.896304 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.895956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-etc-selinux\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.896304 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.895980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:50.896304 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896005    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-cni-netd\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.896304 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9467694-b4aa-4b01-8a10-05dceaebc511-agent-certs\") pod \"konnectivity-agent-zl482\" (UID: \"a9467694-b4aa-4b01-8a10-05dceaebc511\") " pod="kube-system/konnectivity-agent-zl482"
Jul 21 17:25:50.896304 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896051    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-socket-dir\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.896304 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896092    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-var-lib-kubelet\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:50.896304 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896117    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-var-lib-cni-bin\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.896304 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896142    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15-tmp-dir\") pod \"node-resolver-8gvpd\" (UID: \"85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15\") " pod="openshift-dns/node-resolver-8gvpd"
Jul 21 17:25:50.896304 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896165    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-device-dir\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.896304 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896190    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:50.896304 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896215    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-node-log\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.896999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-multus-socket-dir-parent\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.896999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896264    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-multus-conf-dir\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.896999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896286    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5e623c85-720f-4764-9898-3a0ecc93a45a-multus-daemon-config\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.896999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896311    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b35c0345-124e-40c5-8de6-2b9c3122e24a-host\") pod \"node-ca-xxbb8\" (UID: \"b35c0345-124e-40c5-8de6-2b9c3122e24a\") " pod="openshift-image-registry/node-ca-xxbb8"
Jul 21 17:25:50.896999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896344    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wndt4\" (UniqueName: \"kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4\") pod \"network-check-target-r6qp2\" (UID: \"c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4\") " pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:25:50.896999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896370    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svxwl\" (UniqueName: \"kubernetes.io/projected/bbf98769-a9a3-4a65-8a28-33809628690b-kube-api-access-svxwl\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.896999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896392    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-systemd\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:50.896999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896417    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xzk9\" (UniqueName: \"kubernetes.io/projected/4d4cddb0-f0f9-45b2-a606-00eca147717a-kube-api-access-9xzk9\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:50.896999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896460    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-cnibin\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:50.896999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896496    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:50.896999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-ovn-node-metrics-cert\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.896999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896545    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:50.896999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896556    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-socket-dir\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.896999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896570    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-kubelet\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.896999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896608    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-ovnkube-script-lib\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.896999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896644    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-system-cni-dir\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.897734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896754    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-etc-selinux\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.897734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896817    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-device-dir\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.897734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.896892    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b35c0345-124e-40c5-8de6-2b9c3122e24a-host\") pod \"node-ca-xxbb8\" (UID: \"b35c0345-124e-40c5-8de6-2b9c3122e24a\") " pod="openshift-image-registry/node-ca-xxbb8"
Jul 21 17:25:50.897734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897365    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-etc-kubernetes\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.897734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897395    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a016f87b-eabf-47b0-a16f-94dafd80e108-host-slash\") pod \"iptables-alerter-5gtz4\" (UID: \"a016f87b-eabf-47b0-a16f-94dafd80e108\") " pod="openshift-network-operator/iptables-alerter-5gtz4"
Jul 21 17:25:50.897734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897419    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-kubernetes\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:50.897734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897446    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-run-netns\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.897734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897480    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-var-lib-openvswitch\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.897734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.897734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897530    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-cni-bin\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.897734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897554    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-multus-cni-dir\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.897734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897578    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-hostroot\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.897734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897605    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b35c0345-124e-40c5-8de6-2b9c3122e24a-serviceca\") pod \"node-ca-xxbb8\" (UID: \"b35c0345-124e-40c5-8de6-2b9c3122e24a\") " pod="openshift-image-registry/node-ca-xxbb8"
Jul 21 17:25:50.897734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897630    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-sysconfig\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:50.897734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-run\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:50.898479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897746    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15-hosts-file\") pod \"node-resolver-8gvpd\" (UID: \"85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15\") " pod="openshift-dns/node-resolver-8gvpd"
Jul 21 17:25:50.898479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897772    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-sysctl-d\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:50.898479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897797    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-sysctl-conf\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:50.898479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-env-overrides\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.898479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897846    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9467694-b4aa-4b01-8a10-05dceaebc511-konnectivity-ca\") pod \"konnectivity-agent-zl482\" (UID: \"a9467694-b4aa-4b01-8a10-05dceaebc511\") " pod="kube-system/konnectivity-agent-zl482"
Jul 21 17:25:50.898479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897872    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-dbus\") pod \"global-pull-secret-syncer-qht4v\" (UID: \"e5718cfc-649b-4cd1-bd58-0ad5494c5e56\") " pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:50.898479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret\") pod \"global-pull-secret-syncer-qht4v\" (UID: \"e5718cfc-649b-4cd1-bd58-0ad5494c5e56\") " pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:50.898479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897926    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a016f87b-eabf-47b0-a16f-94dafd80e108-iptables-alerter-script\") pod \"iptables-alerter-5gtz4\" (UID: \"a016f87b-eabf-47b0-a16f-94dafd80e108\") " pod="openshift-network-operator/iptables-alerter-5gtz4"
Jul 21 17:25:50.898479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-registration-dir\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.898479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.897981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-sys-fs\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.898479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898009    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqfgf\" (UniqueName: \"kubernetes.io/projected/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-kube-api-access-gqfgf\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:50.898479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898033    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-run-openvswitch\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.898479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-run-systemd\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.898479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898103    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-run-ovn\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.898479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898127    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.898479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898250    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-ovnkube-config\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898274    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-cnibin\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-lib-modules\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898343    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr6gp\" (UniqueName: \"kubernetes.io/projected/b35c0345-124e-40c5-8de6-2b9c3122e24a-kube-api-access-zr6gp\") pod \"node-ca-xxbb8\" (UID: \"b35c0345-124e-40c5-8de6-2b9c3122e24a\") " pod="openshift-image-registry/node-ca-xxbb8"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898370    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898395    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-sys\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898418    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-host\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898441    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-slash\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898468    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-etc-openvswitch\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-kubelet-config\") pod \"global-pull-secret-syncer-qht4v\" (UID: \"e5718cfc-649b-4cd1-bd58-0ad5494c5e56\") " pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898518    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsmvl\" (UniqueName: \"kubernetes.io/projected/5e623c85-720f-4764-9898-3a0ecc93a45a-kube-api-access-dsmvl\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898545    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tskrf\" (UniqueName: \"kubernetes.io/projected/cb83c0b0-6020-43bf-be65-b41d51b76e5a-kube-api-access-tskrf\") pod \"network-metrics-daemon-9r9z4\" (UID: \"cb83c0b0-6020-43bf-be65-b41d51b76e5a\") " pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898577    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-system-cni-dir\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d4cddb0-f0f9-45b2-a606-00eca147717a-tmp\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898625    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-run-k8s-cni-cncf-io\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898649    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-var-lib-cni-multus\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.898675    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-var-lib-kubelet\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.899206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.899184    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b35c0345-124e-40c5-8de6-2b9c3122e24a-serviceca\") pod \"node-ca-xxbb8\" (UID: \"b35c0345-124e-40c5-8de6-2b9c3122e24a\") " pod="openshift-image-registry/node-ca-xxbb8"
Jul 21 17:25:50.899948 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.899277    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-registration-dir\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.899948 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.899325    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-sys-fs\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.899948 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.899764    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bbf98769-a9a3-4a65-8a28-33809628690b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.902791 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.902748    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 21 17:25:50.902953 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:50.902937    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:25:50.903017 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:50.902957    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:25:50.903017 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:50.902966    2562 projected.go:196] Error preparing data for projected volume kube-api-access-wndt4 for pod openshift-network-diagnostics/network-check-target-r6qp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:50.903135 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:50.903029    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4 podName:c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:51.403000039 +0000 UTC m=+2.974621789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wndt4" (UniqueName: "kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4") pod "network-check-target-r6qp2" (UID: "c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:50.906348 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.906326    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svxwl\" (UniqueName: \"kubernetes.io/projected/bbf98769-a9a3-4a65-8a28-33809628690b-kube-api-access-svxwl\") pod \"aws-ebs-csi-driver-node-qddjt\" (UID: \"bbf98769-a9a3-4a65-8a28-33809628690b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:50.907260 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.907240    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr6gp\" (UniqueName: \"kubernetes.io/projected/b35c0345-124e-40c5-8de6-2b9c3122e24a-kube-api-access-zr6gp\") pod \"node-ca-xxbb8\" (UID: \"b35c0345-124e-40c5-8de6-2b9c3122e24a\") " pod="openshift-image-registry/node-ca-xxbb8"
Jul 21 17:25:50.999688 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.999654    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-node-log\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.999688 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.999698    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-multus-socket-dir-parent\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.999927 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.999728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-multus-conf-dir\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.999927 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.999792    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-node-log\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:50.999927 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.999847    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5e623c85-720f-4764-9898-3a0ecc93a45a-multus-daemon-config\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:50.999927 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.999901    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-systemd\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:50.999927 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.999926    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xzk9\" (UniqueName: \"kubernetes.io/projected/4d4cddb0-f0f9-45b2-a606-00eca147717a-kube-api-access-9xzk9\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.000196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.999947    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-cnibin\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.000196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.999965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.000196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:50.999985    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-ovn-node-metrics-cert\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.000196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000010    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.000196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-kubelet\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.000196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000058    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-multus-conf-dir\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.000196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000058    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-multus-socket-dir-parent\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.000196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000077    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-ovnkube-script-lib\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.000196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000128    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-system-cni-dir\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.000196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000151    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-etc-kubernetes\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.000196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a016f87b-eabf-47b0-a16f-94dafd80e108-host-slash\") pod \"iptables-alerter-5gtz4\" (UID: \"a016f87b-eabf-47b0-a16f-94dafd80e108\") " pod="openshift-network-operator/iptables-alerter-5gtz4"
Jul 21 17:25:51.000196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000198    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-kubernetes\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000223    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-run-netns\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000227    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000247    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-var-lib-openvswitch\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-systemd\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000298    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-cni-bin\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000319    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-multus-cni-dir\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000341    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-hostroot\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-sysconfig\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-run\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000404    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15-hosts-file\") pod \"node-resolver-8gvpd\" (UID: \"85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15\") " pod="openshift-dns/node-resolver-8gvpd"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000420    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-cnibin\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000492    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-system-cni-dir\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a016f87b-eabf-47b0-a16f-94dafd80e108-host-slash\") pod \"iptables-alerter-5gtz4\" (UID: \"a016f87b-eabf-47b0-a16f-94dafd80e108\") " pod="openshift-network-operator/iptables-alerter-5gtz4"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000559    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-sysctl-d\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000425    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-sysctl-d\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000584    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5e623c85-720f-4764-9898-3a0ecc93a45a-multus-daemon-config\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.000606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000608    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-sysctl-conf\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000634    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-env-overrides\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000641    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-cni-bin\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000658    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9467694-b4aa-4b01-8a10-05dceaebc511-konnectivity-ca\") pod \"konnectivity-agent-zl482\" (UID: \"a9467694-b4aa-4b01-8a10-05dceaebc511\") " pod="kube-system/konnectivity-agent-zl482"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000682    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-dbus\") pod \"global-pull-secret-syncer-qht4v\" (UID: \"e5718cfc-649b-4cd1-bd58-0ad5494c5e56\") " pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000686    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-kubernetes\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000707    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret\") pod \"global-pull-secret-syncer-qht4v\" (UID: \"e5718cfc-649b-4cd1-bd58-0ad5494c5e56\") " pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000722    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-run-netns\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000732    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a016f87b-eabf-47b0-a16f-94dafd80e108-iptables-alerter-script\") pod \"iptables-alerter-5gtz4\" (UID: \"a016f87b-eabf-47b0-a16f-94dafd80e108\") " pod="openshift-network-operator/iptables-alerter-5gtz4"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000761    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqfgf\" (UniqueName: \"kubernetes.io/projected/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-kube-api-access-gqfgf\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000762    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-var-lib-openvswitch\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000857    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-kubelet\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000897    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-run-openvswitch\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000916    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-multus-cni-dir\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000924    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-run-systemd\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-run-ovn\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000962    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-hostroot\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.001185 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.000979    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001006    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-ovnkube-config\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001021    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-run\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001030    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-cnibin\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001038    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-ovnkube-script-lib\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001038    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001095    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15-hosts-file\") pod \"node-resolver-8gvpd\" (UID: \"85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15\") " pod="openshift-dns/node-resolver-8gvpd"
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001103    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-run-openvswitch\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001114    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-etc-kubernetes\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:51.001117    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001140    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-lib-modules\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001172    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-run-ovn\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:51.001179    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret podName:e5718cfc-649b-4cd1-bd58-0ad5494c5e56 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:51.501160206 +0000 UTC m=+3.072781958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret") pod "global-pull-secret-syncer-qht4v" (UID: "e5718cfc-649b-4cd1-bd58-0ad5494c5e56") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001195    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-sys\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001217    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-run-systemd\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001219    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-host\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001242    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-lib-modules\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.001725 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001253    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-host\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001106    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-sysctl-conf\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001253    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-slash\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001288    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001294    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-slash\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001290    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-etc-openvswitch\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001320    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-etc-openvswitch\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001327    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-kubelet-config\") pod \"global-pull-secret-syncer-qht4v\" (UID: \"e5718cfc-649b-4cd1-bd58-0ad5494c5e56\") " pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001367    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-sys\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsmvl\" (UniqueName: \"kubernetes.io/projected/5e623c85-720f-4764-9898-3a0ecc93a45a-kube-api-access-dsmvl\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001400    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tskrf\" (UniqueName: \"kubernetes.io/projected/cb83c0b0-6020-43bf-be65-b41d51b76e5a-kube-api-access-tskrf\") pod \"network-metrics-daemon-9r9z4\" (UID: \"cb83c0b0-6020-43bf-be65-b41d51b76e5a\") " pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001428    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-system-cni-dir\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001451    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d4cddb0-f0f9-45b2-a606-00eca147717a-tmp\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001475    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-run-k8s-cni-cncf-io\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001500    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-var-lib-cni-multus\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001524    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-var-lib-kubelet\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001552    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-modprobe-d\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.002516 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001577    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-log-socket\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001597    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk62t\" (UniqueName: \"kubernetes.io/projected/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-kube-api-access-tk62t\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001606    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-dbus\") pod \"global-pull-secret-syncer-qht4v\" (UID: \"e5718cfc-649b-4cd1-bd58-0ad5494c5e56\") " pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001616    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9467694-b4aa-4b01-8a10-05dceaebc511-konnectivity-ca\") pod \"konnectivity-agent-zl482\" (UID: \"a9467694-b4aa-4b01-8a10-05dceaebc511\") " pod="kube-system/konnectivity-agent-zl482"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001657    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-kubelet-config\") pod \"global-pull-secret-syncer-qht4v\" (UID: \"e5718cfc-649b-4cd1-bd58-0ad5494c5e56\") " pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001923    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-sysconfig\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.002311    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a016f87b-eabf-47b0-a16f-94dafd80e108-iptables-alerter-script\") pod \"iptables-alerter-5gtz4\" (UID: \"a016f87b-eabf-47b0-a16f-94dafd80e108\") " pod="openshift-network-operator/iptables-alerter-5gtz4"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001667    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-env-overrides\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.002430    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-os-release\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001620    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-os-release\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.001140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-cnibin\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.002499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e623c85-720f-4764-9898-3a0ecc93a45a-cni-binary-copy\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.002519    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-var-lib-cni-multus\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.002527    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-run-netns\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.002546    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-ovnkube-config\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.002615    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-run-k8s-cni-cncf-io\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.002744    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-run-netns\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.002781    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-system-cni-dir\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.003370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.002817    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-log-socket\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.002848    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-var-lib-kubelet\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.002935    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-modprobe-d\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.003132    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e623c85-720f-4764-9898-3a0ecc93a45a-cni-binary-copy\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.003343    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-run-multus-certs\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.003474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-tuned\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.003509    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs\") pod \"network-metrics-daemon-9r9z4\" (UID: \"cb83c0b0-6020-43bf-be65-b41d51b76e5a\") " pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.003541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-os-release\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.003567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-systemd-units\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.003643    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544bm\" (UniqueName: \"kubernetes.io/projected/85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15-kube-api-access-544bm\") pod \"node-resolver-8gvpd\" (UID: \"85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15\") " pod="openshift-dns/node-resolver-8gvpd"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.003675    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7lmb\" (UniqueName: \"kubernetes.io/projected/a016f87b-eabf-47b0-a16f-94dafd80e108-kube-api-access-l7lmb\") pod \"iptables-alerter-5gtz4\" (UID: \"a016f87b-eabf-47b0-a16f-94dafd80e108\") " pod="openshift-network-operator/iptables-alerter-5gtz4"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.003708    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.003804    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-cni-netd\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.003836    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9467694-b4aa-4b01-8a10-05dceaebc511-agent-certs\") pod \"konnectivity-agent-zl482\" (UID: \"a9467694-b4aa-4b01-8a10-05dceaebc511\") " pod="kube-system/konnectivity-agent-zl482"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.003870    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-var-lib-kubelet\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.003915    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-os-release\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.003901    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-var-lib-cni-bin\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.004233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.003990    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-run-multus-certs\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.005003 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.004000    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15-tmp-dir\") pod \"node-resolver-8gvpd\" (UID: \"85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15\") " pod="openshift-dns/node-resolver-8gvpd"
Jul 21 17:25:51.005003 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.004032    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.005003 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:51.004330    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:51.005003 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:51.004839    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs podName:cb83c0b0-6020-43bf-be65-b41d51b76e5a nodeName:}" failed. No retries permitted until 2026-07-21 17:25:51.50475866 +0000 UTC m=+3.076380402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs") pod "network-metrics-daemon-9r9z4" (UID: "cb83c0b0-6020-43bf-be65-b41d51b76e5a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:51.005003 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.004916    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.005003 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.005000    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-systemd-units\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.008373 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.005547    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d4cddb0-f0f9-45b2-a606-00eca147717a-tmp\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.008373 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.005844    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e623c85-720f-4764-9898-3a0ecc93a45a-host-var-lib-cni-bin\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.008373 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.005919    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d4cddb0-f0f9-45b2-a606-00eca147717a-var-lib-kubelet\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.008373 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.006090    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.008373 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.004920    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-host-cni-netd\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.008373 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.006335    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15-tmp-dir\") pod \"node-resolver-8gvpd\" (UID: \"85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15\") " pod="openshift-dns/node-resolver-8gvpd"
Jul 21 17:25:51.008373 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.006495    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-ovn-node-metrics-cert\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.009265 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.008879    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9467694-b4aa-4b01-8a10-05dceaebc511-agent-certs\") pod \"konnectivity-agent-zl482\" (UID: \"a9467694-b4aa-4b01-8a10-05dceaebc511\") " pod="kube-system/konnectivity-agent-zl482"
Jul 21 17:25:51.009378 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.009358    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d4cddb0-f0f9-45b2-a606-00eca147717a-etc-tuned\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.009581 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.009561    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xzk9\" (UniqueName: \"kubernetes.io/projected/4d4cddb0-f0f9-45b2-a606-00eca147717a-kube-api-access-9xzk9\") pod \"tuned-rkbgr\" (UID: \"4d4cddb0-f0f9-45b2-a606-00eca147717a\") " pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.010404 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.010168    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqfgf\" (UniqueName: \"kubernetes.io/projected/7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1-kube-api-access-gqfgf\") pod \"multus-additional-cni-plugins-7rnxt\" (UID: \"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1\") " pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.012558 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.012536    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsmvl\" (UniqueName: \"kubernetes.io/projected/5e623c85-720f-4764-9898-3a0ecc93a45a-kube-api-access-dsmvl\") pod \"multus-tx58r\" (UID: \"5e623c85-720f-4764-9898-3a0ecc93a45a\") " pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.013559 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.013538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-544bm\" (UniqueName: \"kubernetes.io/projected/85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15-kube-api-access-544bm\") pod \"node-resolver-8gvpd\" (UID: \"85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15\") " pod="openshift-dns/node-resolver-8gvpd"
Jul 21 17:25:51.013879 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.013860    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7lmb\" (UniqueName: \"kubernetes.io/projected/a016f87b-eabf-47b0-a16f-94dafd80e108-kube-api-access-l7lmb\") pod \"iptables-alerter-5gtz4\" (UID: \"a016f87b-eabf-47b0-a16f-94dafd80e108\") " pod="openshift-network-operator/iptables-alerter-5gtz4"
Jul 21 17:25:51.014165 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.014116    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tskrf\" (UniqueName: \"kubernetes.io/projected/cb83c0b0-6020-43bf-be65-b41d51b76e5a-kube-api-access-tskrf\") pod \"network-metrics-daemon-9r9z4\" (UID: \"cb83c0b0-6020-43bf-be65-b41d51b76e5a\") " pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:25:51.014261 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.014229    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk62t\" (UniqueName: \"kubernetes.io/projected/0c36d72b-9bf6-4d72-a2b8-84451278a3d2-kube-api-access-tk62t\") pod \"ovnkube-node-bdplm\" (UID: \"0c36d72b-9bf6-4d72-a2b8-84451278a3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.091385 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.091294    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xxbb8"
Jul 21 17:25:51.101412 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.101387    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt"
Jul 21 17:25:51.108872 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.108848    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rkbgr"
Jul 21 17:25:51.115618 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.115586    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zl482"
Jul 21 17:25:51.123155 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.123136    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7rnxt"
Jul 21 17:25:51.129860 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.129837    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tx58r"
Jul 21 17:25:51.135666 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.135642    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8gvpd"
Jul 21 17:25:51.143274 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.143255    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5gtz4"
Jul 21 17:25:51.149989 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.149971    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:25:51.318462 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.318433    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:25:51.408858 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.408828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wndt4\" (UniqueName: \"kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4\") pod \"network-check-target-r6qp2\" (UID: \"c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4\") " pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:25:51.409014 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:51.408996    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:25:51.409109 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:51.409025    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:25:51.409109 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:51.409038    2562 projected.go:196] Error preparing data for projected volume kube-api-access-wndt4 for pod openshift-network-diagnostics/network-check-target-r6qp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:51.409215 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:51.409131    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4 podName:c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:52.409109391 +0000 UTC m=+3.980731146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wndt4" (UniqueName: "kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4") pod "network-check-target-r6qp2" (UID: "c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:51.509942 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.509911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs\") pod \"network-metrics-daemon-9r9z4\" (UID: \"cb83c0b0-6020-43bf-be65-b41d51b76e5a\") " pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:25:51.510146 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.510005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret\") pod \"global-pull-secret-syncer-qht4v\" (UID: \"e5718cfc-649b-4cd1-bd58-0ad5494c5e56\") " pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:51.510146 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:51.510107    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:51.510146 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:51.510132    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:51.510271 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:51.510180    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs podName:cb83c0b0-6020-43bf-be65-b41d51b76e5a nodeName:}" failed. No retries permitted until 2026-07-21 17:25:52.510158637 +0000 UTC m=+4.081780389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs") pod "network-metrics-daemon-9r9z4" (UID: "cb83c0b0-6020-43bf-be65-b41d51b76e5a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:51.510271 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:51.510201    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret podName:e5718cfc-649b-4cd1-bd58-0ad5494c5e56 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:52.510189998 +0000 UTC m=+4.081811735 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret") pod "global-pull-secret-syncer-qht4v" (UID: "e5718cfc-649b-4cd1-bd58-0ad5494c5e56") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:51.705763 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:51.705719    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c36d72b_9bf6_4d72_a2b8_84451278a3d2.slice/crio-b4da2a8ea9356feac7a00f3db622b7238652457710e33b7a974a95f017a40d3b WatchSource:0}: Error finding container b4da2a8ea9356feac7a00f3db622b7238652457710e33b7a974a95f017a40d3b: Status 404 returned error can't find the container with id b4da2a8ea9356feac7a00f3db622b7238652457710e33b7a974a95f017a40d3b
Jul 21 17:25:51.708645 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:51.708624    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e623c85_720f_4764_9898_3a0ecc93a45a.slice/crio-c9166374da00e59b4545f131f86657ef143f4adf08352f587172d9d251c00be7 WatchSource:0}: Error finding container c9166374da00e59b4545f131f86657ef143f4adf08352f587172d9d251c00be7: Status 404 returned error can't find the container with id c9166374da00e59b4545f131f86657ef143f4adf08352f587172d9d251c00be7
Jul 21 17:25:51.710579 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:51.710553    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85fe9fd8_3fbd_4e4a_9300_b0c6c117fe15.slice/crio-80614fa01578869802fc5200a1ddd0ac7f50f5ce1397046eb697aacc2525e40e WatchSource:0}: Error finding container 80614fa01578869802fc5200a1ddd0ac7f50f5ce1397046eb697aacc2525e40e: Status 404 returned error can't find the container with id 80614fa01578869802fc5200a1ddd0ac7f50f5ce1397046eb697aacc2525e40e
Jul 21 17:25:51.712909 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:51.712886    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe18fe3_d1e6_4f34_a6b7_c1db7bafc4f1.slice/crio-fb7857786463b924d36f802bda46b349fb82c3d731ac1acf7929af6a2d8899be WatchSource:0}: Error finding container fb7857786463b924d36f802bda46b349fb82c3d731ac1acf7929af6a2d8899be: Status 404 returned error can't find the container with id fb7857786463b924d36f802bda46b349fb82c3d731ac1acf7929af6a2d8899be
Jul 21 17:25:51.734329 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:51.734305    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf98769_a9a3_4a65_8a28_33809628690b.slice/crio-eff2443a934c849e1e5239209d8b36104e68aa68732ce65a02b4c34807a3a91d WatchSource:0}: Error finding container eff2443a934c849e1e5239209d8b36104e68aa68732ce65a02b4c34807a3a91d: Status 404 returned error can't find the container with id eff2443a934c849e1e5239209d8b36104e68aa68732ce65a02b4c34807a3a91d
Jul 21 17:25:51.735469 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:51.735447    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35c0345_124e_40c5_8de6_2b9c3122e24a.slice/crio-0d09701fa89efdd41e78af56fe52e122e1b7279088ab9be3d9af3e472a9ec99b WatchSource:0}: Error finding container 0d09701fa89efdd41e78af56fe52e122e1b7279088ab9be3d9af3e472a9ec99b: Status 404 returned error can't find the container with id 0d09701fa89efdd41e78af56fe52e122e1b7279088ab9be3d9af3e472a9ec99b
Jul 21 17:25:51.737246 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:51.736780    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda016f87b_eabf_47b0_a16f_94dafd80e108.slice/crio-d14361e0e56778b08c2e0b5e3fb74c4d9a9a7e1f23e1387aa04c6e074a33f48f WatchSource:0}: Error finding container d14361e0e56778b08c2e0b5e3fb74c4d9a9a7e1f23e1387aa04c6e074a33f48f: Status 404 returned error can't find the container with id d14361e0e56778b08c2e0b5e3fb74c4d9a9a7e1f23e1387aa04c6e074a33f48f
Jul 21 17:25:51.739134 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:51.738139    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4cddb0_f0f9_45b2_a606_00eca147717a.slice/crio-946204925af7996a8985078e30bf80818f3536b65f32dc767431f18ed4cbf0a4 WatchSource:0}: Error finding container 946204925af7996a8985078e30bf80818f3536b65f32dc767431f18ed4cbf0a4: Status 404 returned error can't find the container with id 946204925af7996a8985078e30bf80818f3536b65f32dc767431f18ed4cbf0a4
Jul 21 17:25:51.739134 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:25:51.738610    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9467694_b4aa_4b01_8a10_05dceaebc511.slice/crio-73a234c53af62aadd005bbfc4b9619f6d6c062a708e6a7d9f7a19ee6288ef147 WatchSource:0}: Error finding container 73a234c53af62aadd005bbfc4b9619f6d6c062a708e6a7d9f7a19ee6288ef147: Status 404 returned error can't find the container with id 73a234c53af62aadd005bbfc4b9619f6d6c062a708e6a7d9f7a19ee6288ef147
Jul 21 17:25:51.828640 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.828481    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 17:20:50 +0000 UTC" deadline="2028-01-24 02:45:08.054064895 +0000 UTC"
Jul 21 17:25:51.829012 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.828644    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13233h19m16.225427528s"
Jul 21 17:25:51.923857 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.923827    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rkbgr" event={"ID":"4d4cddb0-f0f9-45b2-a606-00eca147717a","Type":"ContainerStarted","Data":"946204925af7996a8985078e30bf80818f3536b65f32dc767431f18ed4cbf0a4"}
Jul 21 17:25:51.924999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.924976    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5gtz4" event={"ID":"a016f87b-eabf-47b0-a16f-94dafd80e108","Type":"ContainerStarted","Data":"d14361e0e56778b08c2e0b5e3fb74c4d9a9a7e1f23e1387aa04c6e074a33f48f"}
Jul 21 17:25:51.925907 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.925885    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xxbb8" event={"ID":"b35c0345-124e-40c5-8de6-2b9c3122e24a","Type":"ContainerStarted","Data":"0d09701fa89efdd41e78af56fe52e122e1b7279088ab9be3d9af3e472a9ec99b"}
Jul 21 17:25:51.927388 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.927360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8gvpd" event={"ID":"85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15","Type":"ContainerStarted","Data":"80614fa01578869802fc5200a1ddd0ac7f50f5ce1397046eb697aacc2525e40e"}
Jul 21 17:25:51.929012 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.928985    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm" event={"ID":"0c36d72b-9bf6-4d72-a2b8-84451278a3d2","Type":"ContainerStarted","Data":"b4da2a8ea9356feac7a00f3db622b7238652457710e33b7a974a95f017a40d3b"}
Jul 21 17:25:51.931363 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.931341    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-228.ec2.internal" event={"ID":"5c1883a5f27b82c72d9daa1f8b8a95db","Type":"ContainerStarted","Data":"b5b24a12d20a5281af8d2df25b550d98dcdf2f01cdfdc6dc85d6ce6a5e876dc1"}
Jul 21 17:25:51.932630 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.932610    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rnxt" event={"ID":"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1","Type":"ContainerStarted","Data":"fb7857786463b924d36f802bda46b349fb82c3d731ac1acf7929af6a2d8899be"}
Jul 21 17:25:51.933616 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.933598    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tx58r" event={"ID":"5e623c85-720f-4764-9898-3a0ecc93a45a","Type":"ContainerStarted","Data":"c9166374da00e59b4545f131f86657ef143f4adf08352f587172d9d251c00be7"}
Jul 21 17:25:51.934630 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.934551    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zl482" event={"ID":"a9467694-b4aa-4b01-8a10-05dceaebc511","Type":"ContainerStarted","Data":"73a234c53af62aadd005bbfc4b9619f6d6c062a708e6a7d9f7a19ee6288ef147"}
Jul 21 17:25:51.936007 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.935989    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt" event={"ID":"bbf98769-a9a3-4a65-8a28-33809628690b","Type":"ContainerStarted","Data":"eff2443a934c849e1e5239209d8b36104e68aa68732ce65a02b4c34807a3a91d"}
Jul 21 17:25:51.946966 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:51.946881    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-228.ec2.internal" podStartSLOduration=2.946867309 podStartE2EDuration="2.946867309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:25:51.946691115 +0000 UTC m=+3.518312869" watchObservedRunningTime="2026-07-21 17:25:51.946867309 +0000 UTC m=+3.518489064"
Jul 21 17:25:52.417359 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:52.417326    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wndt4\" (UniqueName: \"kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4\") pod \"network-check-target-r6qp2\" (UID: \"c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4\") " pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:25:52.417558 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:52.417542    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:25:52.417623 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:52.417565    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:25:52.417623 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:52.417578    2562 projected.go:196] Error preparing data for projected volume kube-api-access-wndt4 for pod openshift-network-diagnostics/network-check-target-r6qp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:52.417718 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:52.417639    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4 podName:c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:54.417620468 +0000 UTC m=+5.989242221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wndt4" (UniqueName: "kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4") pod "network-check-target-r6qp2" (UID: "c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:52.518471 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:52.517698    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs\") pod \"network-metrics-daemon-9r9z4\" (UID: \"cb83c0b0-6020-43bf-be65-b41d51b76e5a\") " pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:25:52.518471 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:52.517786    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret\") pod \"global-pull-secret-syncer-qht4v\" (UID: \"e5718cfc-649b-4cd1-bd58-0ad5494c5e56\") " pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:52.518471 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:52.517905    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:52.518471 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:52.517987    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret podName:e5718cfc-649b-4cd1-bd58-0ad5494c5e56 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:54.517949792 +0000 UTC m=+6.089571528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret") pod "global-pull-secret-syncer-qht4v" (UID: "e5718cfc-649b-4cd1-bd58-0ad5494c5e56") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:52.518471 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:52.518378    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:52.518471 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:52.518432    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs podName:cb83c0b0-6020-43bf-be65-b41d51b76e5a nodeName:}" failed. No retries permitted until 2026-07-21 17:25:54.518413717 +0000 UTC m=+6.090035536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs") pod "network-metrics-daemon-9r9z4" (UID: "cb83c0b0-6020-43bf-be65-b41d51b76e5a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:52.829711 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:52.829564    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 17:20:50 +0000 UTC" deadline="2028-02-10 09:18:34.584688988 +0000 UTC"
Jul 21 17:25:52.829711 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:52.829605    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13647h52m41.755089986s"
Jul 21 17:25:52.918298 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:52.918266    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:25:52.918460 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:52.918410    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:25:52.918802 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:52.918783    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:52.918906 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:52.918880    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:25:52.918969 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:52.918942    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:25:52.919028 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:52.919009    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:25:52.948016 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:52.947984    2562 generic.go:356] "Generic (PLEG): container finished" podID="91cb1fde35779c2f274066a1ceeb532c" containerID="438a24aa469c28615302619d5120fbf11d857c551a0e683bb411aaa36b771928" exitCode=0
Jul 21 17:25:52.948960 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:52.948932    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal" event={"ID":"91cb1fde35779c2f274066a1ceeb532c","Type":"ContainerDied","Data":"438a24aa469c28615302619d5120fbf11d857c551a0e683bb411aaa36b771928"}
Jul 21 17:25:53.962770 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:53.962724    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal" event={"ID":"91cb1fde35779c2f274066a1ceeb532c","Type":"ContainerStarted","Data":"a03b702598e2cbfce827019607c7d233a1bc0785ae6805d554163cb339c556d6"}
Jul 21 17:25:53.983008 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:53.982181    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-228.ec2.internal" podStartSLOduration=4.98215651 podStartE2EDuration="4.98215651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:25:53.981743944 +0000 UTC m=+5.553365702" watchObservedRunningTime="2026-07-21 17:25:53.98215651 +0000 UTC m=+5.553778266"
Jul 21 17:25:54.435857 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:54.435816    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wndt4\" (UniqueName: \"kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4\") pod \"network-check-target-r6qp2\" (UID: \"c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4\") " pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:25:54.436029 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:54.436011    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:25:54.436109 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:54.436040    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:25:54.436109 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:54.436054    2562 projected.go:196] Error preparing data for projected volume kube-api-access-wndt4 for pod openshift-network-diagnostics/network-check-target-r6qp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:54.436194 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:54.436128    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4 podName:c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:58.436108791 +0000 UTC m=+10.007730528 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wndt4" (UniqueName: "kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4") pod "network-check-target-r6qp2" (UID: "c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:54.536845 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:54.536805    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret\") pod \"global-pull-secret-syncer-qht4v\" (UID: \"e5718cfc-649b-4cd1-bd58-0ad5494c5e56\") " pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:54.537036 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:54.536877    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs\") pod \"network-metrics-daemon-9r9z4\" (UID: \"cb83c0b0-6020-43bf-be65-b41d51b76e5a\") " pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:25:54.537036 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:54.537013    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:54.537232 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:54.537094    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs podName:cb83c0b0-6020-43bf-be65-b41d51b76e5a nodeName:}" failed. No retries permitted until 2026-07-21 17:25:58.537056795 +0000 UTC m=+10.108678534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs") pod "network-metrics-daemon-9r9z4" (UID: "cb83c0b0-6020-43bf-be65-b41d51b76e5a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:54.537506 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:54.537478    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:54.537584 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:54.537544    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret podName:e5718cfc-649b-4cd1-bd58-0ad5494c5e56 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:58.537527372 +0000 UTC m=+10.109149108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret") pod "global-pull-secret-syncer-qht4v" (UID: "e5718cfc-649b-4cd1-bd58-0ad5494c5e56") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:54.917204 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:54.917171    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:25:54.917386 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:54.917287    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:25:54.917578 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:54.917547    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:54.917679 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:54.917662    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:25:54.917743 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:54.917725    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:25:54.917830 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:54.917811    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:25:56.916216 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:56.916184    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:56.916657 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:56.916186    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:25:56.916657 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:56.916318    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:25:56.916657 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:56.916195    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:25:56.916657 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:56.916451    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:25:56.916657 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:56.916534    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:25:58.474413 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:58.474375    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wndt4\" (UniqueName: \"kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4\") pod \"network-check-target-r6qp2\" (UID: \"c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4\") " pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:25:58.474800 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:58.474596    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:25:58.474800 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:58.474618    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:25:58.474800 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:58.474628    2562 projected.go:196] Error preparing data for projected volume kube-api-access-wndt4 for pod openshift-network-diagnostics/network-check-target-r6qp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:58.474800 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:58.474685    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4 podName:c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:06.474666359 +0000 UTC m=+18.046288095 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wndt4" (UniqueName: "kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4") pod "network-check-target-r6qp2" (UID: "c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:58.575378 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:58.575338    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs\") pod \"network-metrics-daemon-9r9z4\" (UID: \"cb83c0b0-6020-43bf-be65-b41d51b76e5a\") " pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:25:58.575551 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:58.575434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret\") pod \"global-pull-secret-syncer-qht4v\" (UID: \"e5718cfc-649b-4cd1-bd58-0ad5494c5e56\") " pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:58.575600 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:58.575582    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:58.576145 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:58.575648    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret podName:e5718cfc-649b-4cd1-bd58-0ad5494c5e56 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:06.575628611 +0000 UTC m=+18.147250347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret") pod "global-pull-secret-syncer-qht4v" (UID: "e5718cfc-649b-4cd1-bd58-0ad5494c5e56") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:58.576145 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:58.575720    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:58.576145 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:58.575760    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs podName:cb83c0b0-6020-43bf-be65-b41d51b76e5a nodeName:}" failed. No retries permitted until 2026-07-21 17:26:06.575746662 +0000 UTC m=+18.147368398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs") pod "network-metrics-daemon-9r9z4" (UID: "cb83c0b0-6020-43bf-be65-b41d51b76e5a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:58.917536 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:58.917505    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:25:58.917698 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:58.917614    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:25:58.917698 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:58.917684    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:25:58.917829 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:58.917749    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:25:58.917829 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:25:58.917803    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:25:58.917931 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:25:58.917869    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:26:00.916581 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:00.916547    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:00.917051 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:00.916668    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:26:00.917051 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:00.916553    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:00.917051 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:00.916789    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:26:00.917051 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:00.916547    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:26:00.917051 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:00.916874    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:26:02.919018 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:02.918988    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:26:02.919018 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:02.919005    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:02.919475 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:02.919005    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:02.919475 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:02.919141    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:26:02.919475 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:02.919229    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:26:02.919475 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:02.919324    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:26:04.918547 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:04.918513    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:04.918547 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:04.918535    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:04.919109 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:04.918535    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:26:04.919109 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:04.918629    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:26:04.919109 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:04.918709    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:26:04.919109 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:04.918838    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:26:06.537995 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:06.537962    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wndt4\" (UniqueName: \"kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4\") pod \"network-check-target-r6qp2\" (UID: \"c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4\") " pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:06.538398 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:06.538145    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:06.538398 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:06.538165    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:06.538398 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:06.538175    2562 projected.go:196] Error preparing data for projected volume kube-api-access-wndt4 for pod openshift-network-diagnostics/network-check-target-r6qp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:06.538398 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:06.538226    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4 podName:c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:22.538212599 +0000 UTC m=+34.109834332 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wndt4" (UniqueName: "kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4") pod "network-check-target-r6qp2" (UID: "c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:06.638794 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:06.638747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs\") pod \"network-metrics-daemon-9r9z4\" (UID: \"cb83c0b0-6020-43bf-be65-b41d51b76e5a\") " pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:26:06.638959 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:06.638845    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret\") pod \"global-pull-secret-syncer-qht4v\" (UID: \"e5718cfc-649b-4cd1-bd58-0ad5494c5e56\") " pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:06.638959 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:06.638900    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:06.639027 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:06.638965    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs podName:cb83c0b0-6020-43bf-be65-b41d51b76e5a nodeName:}" failed. No retries permitted until 2026-07-21 17:26:22.638949018 +0000 UTC m=+34.210570754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs") pod "network-metrics-daemon-9r9z4" (UID: "cb83c0b0-6020-43bf-be65-b41d51b76e5a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:06.639027 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:06.638971    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:06.639027 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:06.639024    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret podName:e5718cfc-649b-4cd1-bd58-0ad5494c5e56 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:22.639007344 +0000 UTC m=+34.210629080 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret") pod "global-pull-secret-syncer-qht4v" (UID: "e5718cfc-649b-4cd1-bd58-0ad5494c5e56") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:06.916594 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:06.916565    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:06.916809 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:06.916697    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:26:06.916809 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:06.916565    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:06.916809 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:06.916797    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:26:06.916912 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:06.916882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:26:06.916960 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:06.916936    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:26:08.916737 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:08.916703    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:08.917246 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:08.916804    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:26:08.917246 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:08.916899    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:26:08.917246 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:08.917025    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:26:08.917246 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:08.917097    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:08.917246 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:08.917172    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:26:10.918856 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:10.918254    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:10.918856 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:10.918381    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:26:10.918856 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:10.918412    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:10.918856 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:10.918484    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:26:10.918856 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:10.918730    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:26:10.918856 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:10.918800    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:26:11.012111 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:11.011906    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rkbgr" event={"ID":"4d4cddb0-f0f9-45b2-a606-00eca147717a","Type":"ContainerStarted","Data":"881ba47b1a319f097a0d8089359fa9585fd655839f6fc1338c59be408271ae5e"}
Jul 21 17:26:11.014099 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:11.014017    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xxbb8" event={"ID":"b35c0345-124e-40c5-8de6-2b9c3122e24a","Type":"ContainerStarted","Data":"94ef975359067bf540c1fa303e801813114c9e416e2544349d2b0eab118ed51a"}
Jul 21 17:26:11.021270 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:11.021097    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8gvpd" event={"ID":"85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15","Type":"ContainerStarted","Data":"5442ef22ad7ad238600349302142414acee4ce4e3e5817b8d3af7b306a6c2d84"}
Jul 21 17:26:11.023030 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:11.023009    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm" event={"ID":"0c36d72b-9bf6-4d72-a2b8-84451278a3d2","Type":"ContainerStarted","Data":"c721a033068b048164b4190d30768a1b9a34b3444a7ac2c1c6d23073ae2a2895"}
Jul 21 17:26:11.024977 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:11.024954    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rnxt" event={"ID":"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1","Type":"ContainerStarted","Data":"f28ea42105d2109147820434157008c0d172e8fdf5cc49b04ca0436252e9f0d2"}
Jul 21 17:26:11.026948 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:11.026895    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tx58r" event={"ID":"5e623c85-720f-4764-9898-3a0ecc93a45a","Type":"ContainerStarted","Data":"6cfaf90133c3cce9a3640e115d25170f3ec98784fdc1c3a30cac83cc0278cea0"}
Jul 21 17:26:11.030778 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:11.030747    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zl482" event={"ID":"a9467694-b4aa-4b01-8a10-05dceaebc511","Type":"ContainerStarted","Data":"967c81674f7033444b6629849c870f015311e6608c62d3ac2d9898e65688fb2f"}
Jul 21 17:26:11.031687 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:11.031645    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rkbgr" podStartSLOduration=3.9784811380000003 podStartE2EDuration="23.031629949s" totalImagesPullingTime="19.053148811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:11.030937239 +0000 UTC m=+22.602558995" watchObservedRunningTime="2026-07-21 17:26:11.031629949 +0000 UTC m=+22.603251707"
Jul 21 17:26:11.033096 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:11.033036    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt" event={"ID":"bbf98769-a9a3-4a65-8a28-33809628690b","Type":"ContainerStarted","Data":"f26c271fc50c2cc24cf632cd2af999414bde0752d899843a6596df25bd6fd739"}
Jul 21 17:26:11.052312 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:11.052269    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tx58r" podStartSLOduration=2.91599931 podStartE2EDuration="22.052251896s" totalImagesPullingTime="19.136252586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:11.051897981 +0000 UTC m=+22.623519759" watchObservedRunningTime="2026-07-21 17:26:11.052251896 +0000 UTC m=+22.623873651"
Jul 21 17:26:11.067554 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:11.067497    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8gvpd" podStartSLOduration=3.008393276 podStartE2EDuration="22.067479823s" totalImagesPullingTime="19.059086547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:11.06704235 +0000 UTC m=+22.638664109" watchObservedRunningTime="2026-07-21 17:26:11.067479823 +0000 UTC m=+22.639101578"
Jul 21 17:26:11.083507 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:11.083454    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xxbb8" podStartSLOduration=4.030784606 podStartE2EDuration="23.083439733s" totalImagesPullingTime="19.052655127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:11.083368855 +0000 UTC m=+22.654990610" watchObservedRunningTime="2026-07-21 17:26:11.083439733 +0000 UTC m=+22.655061487"
Jul 21 17:26:11.130668 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:11.130605    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zl482" podStartSLOduration=11.192406541 podStartE2EDuration="22.130583367s" totalImagesPullingTime="10.938176826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:11.130132814 +0000 UTC m=+22.701754569" watchObservedRunningTime="2026-07-21 17:26:11.130583367 +0000 UTC m=+22.702205144"
Jul 21 17:26:12.038200 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:12.037968    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm" event={"ID":"0c36d72b-9bf6-4d72-a2b8-84451278a3d2","Type":"ContainerStarted","Data":"2fb2c1d67de237daf3c427a24642e1a04aa0d6dfa25dddf05796de0f0b726a27"}
Jul 21 17:26:12.039029 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:12.038245    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm" event={"ID":"0c36d72b-9bf6-4d72-a2b8-84451278a3d2","Type":"ContainerStarted","Data":"2fc890f798c1de06e866646f6b60e16140c056e7933acd28f0c7f114c9ff83ab"}
Jul 21 17:26:12.039029 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:12.038277    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm" event={"ID":"0c36d72b-9bf6-4d72-a2b8-84451278a3d2","Type":"ContainerStarted","Data":"eb5367d2066637f2f593d2cbb2e94bf191396ef87b55f336772ab1e76c21c097"}
Jul 21 17:26:12.039029 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:12.038304    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm" event={"ID":"0c36d72b-9bf6-4d72-a2b8-84451278a3d2","Type":"ContainerStarted","Data":"5dd51c2b0136a4217da4fcc6510bc02df3058fe12314837f780a4d37983f9502"}
Jul 21 17:26:12.039029 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:12.038336    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm" event={"ID":"0c36d72b-9bf6-4d72-a2b8-84451278a3d2","Type":"ContainerStarted","Data":"12e57eda10230396c95f80fef6d68cf557013636126fd7605f3bf7338170f12b"}
Jul 21 17:26:12.039352 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:12.039331    2562 generic.go:356] "Generic (PLEG): container finished" podID="7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1" containerID="f28ea42105d2109147820434157008c0d172e8fdf5cc49b04ca0436252e9f0d2" exitCode=0
Jul 21 17:26:12.039434 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:12.039412    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rnxt" event={"ID":"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1","Type":"ContainerDied","Data":"f28ea42105d2109147820434157008c0d172e8fdf5cc49b04ca0436252e9f0d2"}
Jul 21 17:26:12.040768 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:12.040751    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5gtz4" event={"ID":"a016f87b-eabf-47b0-a16f-94dafd80e108","Type":"ContainerStarted","Data":"44bdc1e0ee34f54c59fa764be8b9549f575b8b4e1731b6eda9c7c03f7e35a41f"}
Jul 21 17:26:12.081676 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:12.081588    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5gtz4" podStartSLOduration=4.032755594 podStartE2EDuration="23.081573744s" totalImagesPullingTime="19.04881815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:12.081243466 +0000 UTC m=+23.652865232" watchObservedRunningTime="2026-07-21 17:26:12.081573744 +0000 UTC m=+23.653195531"
Jul 21 17:26:12.240442 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:12.240415    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 21 17:26:12.853257 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:12.853137    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T17:26:12.240436803Z","UUID":"0a7b2c9b-c3d7-459b-833b-0e30911a83ef","Handler":null,"Name":"","Endpoint":""}
Jul 21 17:26:12.854916 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:12.854890    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 21 17:26:12.855047 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:12.854938    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 21 17:26:12.918224 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:12.918194    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:12.918393 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:12.918194    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:12.918393 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:12.918322    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:26:12.918393 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:12.918385    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:26:12.918502 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:12.918194    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:26:12.918502 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:12.918466    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:26:13.046191 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:13.046153    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt" event={"ID":"bbf98769-a9a3-4a65-8a28-33809628690b","Type":"ContainerStarted","Data":"3cd8c7447296cf3912be7dfe5268ed7a116cd3e8426a6097b7bddbba68326cd7"}
Jul 21 17:26:13.871521 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:13.871485    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zl482"
Jul 21 17:26:13.872217 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:13.872189    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zl482"
Jul 21 17:26:14.051092 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:14.051041    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt" event={"ID":"bbf98769-a9a3-4a65-8a28-33809628690b","Type":"ContainerStarted","Data":"5d3264ed0d65fe98ac07ec419bb3ce5d7f71bb1b0b9fa20edead217f6f920a3e"}
Jul 21 17:26:14.055123 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:14.055091    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm" event={"ID":"0c36d72b-9bf6-4d72-a2b8-84451278a3d2","Type":"ContainerStarted","Data":"ef28e7d4f9ee82d40d1f790d396bfd6bdfc64270993d298b7914a142150e0fd0"}
Jul 21 17:26:14.055329 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:14.055316    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zl482"
Jul 21 17:26:14.055754 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:14.055736    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zl482"
Jul 21 17:26:14.095828 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:14.095767    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qddjt" podStartSLOduration=4.7733012729999995 podStartE2EDuration="26.095749133s" totalImagesPullingTime="21.32244786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:14.074181521 +0000 UTC m=+25.645803287" watchObservedRunningTime="2026-07-21 17:26:14.095749133 +0000 UTC m=+25.667370888"
Jul 21 17:26:14.916462 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:14.916421    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:14.916621 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:14.916427    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:14.916621 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:14.916561    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:26:14.916708 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:14.916635    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:26:14.916708 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:14.916438    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:26:14.916793 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:14.916725    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:26:16.916733 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:16.916555    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:16.917511 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:16.916563    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:26:16.917511 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:16.916814    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:26:16.917511 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:16.916926    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:26:16.917511 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:16.916563    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:16.917511 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:16.917022    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:26:17.062325 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:17.062285    2562 generic.go:356] "Generic (PLEG): container finished" podID="7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1" containerID="fa526d5d99411a62916dd3285e18223ef2ee4174342b6888ea1b255524939978" exitCode=0
Jul 21 17:26:17.062511 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:17.062374    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rnxt" event={"ID":"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1","Type":"ContainerDied","Data":"fa526d5d99411a62916dd3285e18223ef2ee4174342b6888ea1b255524939978"}
Jul 21 17:26:17.066337 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:17.066312    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm" event={"ID":"0c36d72b-9bf6-4d72-a2b8-84451278a3d2","Type":"ContainerStarted","Data":"f4e42f51fa55690301ac809a73b443d02f2b7c66703f2b1c0188fbdd5a5b10ab"}
Jul 21 17:26:17.066520 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:17.066504    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:26:17.066630 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:17.066543    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:26:17.082961 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:17.082937    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:26:17.148513 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:17.148457    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm" podStartSLOduration=8.992368543 podStartE2EDuration="28.148443355s" totalImagesPullingTime="19.156074812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:17.145485069 +0000 UTC m=+28.717106816" watchObservedRunningTime="2026-07-21 17:26:17.148443355 +0000 UTC m=+28.720065088"
Jul 21 17:26:18.069809 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:18.069645    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:26:18.084898 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:18.084874    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:26:18.516422 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:18.516383    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9r9z4"]
Jul 21 17:26:18.516642 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:18.516536    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:26:18.516712 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:18.516681    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:26:18.517045 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:18.517024    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qht4v"]
Jul 21 17:26:18.517144 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:18.517121    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:18.517204 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:18.517187    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:26:18.518568 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:18.518548    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r6qp2"]
Jul 21 17:26:18.518656 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:18.518636    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:18.518743 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:18.518723    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:26:19.073650 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:19.073618    2562 generic.go:356] "Generic (PLEG): container finished" podID="7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1" containerID="5bfeb2893f8a052c966f28604114ea6bbac816759a1f127c456950f448cbbc74" exitCode=0
Jul 21 17:26:19.074127 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:19.073709    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rnxt" event={"ID":"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1","Type":"ContainerDied","Data":"5bfeb2893f8a052c966f28604114ea6bbac816759a1f127c456950f448cbbc74"}
Jul 21 17:26:19.916196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:19.916165    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:26:19.916353 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:19.916302    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:26:19.916417 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:19.916348    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:19.916494 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:19.916471    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:26:20.915751 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:20.915714    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:20.916241 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:20.915827    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:26:21.079805 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:21.079771    2562 generic.go:356] "Generic (PLEG): container finished" podID="7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1" containerID="bbcb921d1d72e911a9c791aa84d9deb409050e650352fcd08fa1d990a3001c9b" exitCode=0
Jul 21 17:26:21.079946 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:21.079833    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rnxt" event={"ID":"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1","Type":"ContainerDied","Data":"bbcb921d1d72e911a9c791aa84d9deb409050e650352fcd08fa1d990a3001c9b"}
Jul 21 17:26:21.915918 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:21.915717    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:26:21.916317 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:21.915760    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:21.916317 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:21.916021    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9r9z4" podUID="cb83c0b0-6020-43bf-be65-b41d51b76e5a"
Jul 21 17:26:21.916317 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:21.916115    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6qp2" podUID="c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4"
Jul 21 17:26:22.559347 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:22.559313    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wndt4\" (UniqueName: \"kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4\") pod \"network-check-target-r6qp2\" (UID: \"c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4\") " pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:22.559537 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:22.559514    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:22.559586 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:22.559545    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:22.559586 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:22.559559    2562 projected.go:196] Error preparing data for projected volume kube-api-access-wndt4 for pod openshift-network-diagnostics/network-check-target-r6qp2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:22.559683 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:22.559624    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4 podName:c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:54.559604601 +0000 UTC m=+66.131226340 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wndt4" (UniqueName: "kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4") pod "network-check-target-r6qp2" (UID: "c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:22.659813 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:22.659778    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret\") pod \"global-pull-secret-syncer-qht4v\" (UID: \"e5718cfc-649b-4cd1-bd58-0ad5494c5e56\") " pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:22.659984 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:22.659829    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs\") pod \"network-metrics-daemon-9r9z4\" (UID: \"cb83c0b0-6020-43bf-be65-b41d51b76e5a\") " pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:26:22.659984 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:22.659932    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:22.659984 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:22.659943    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:22.660176 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:22.660003    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs podName:cb83c0b0-6020-43bf-be65-b41d51b76e5a nodeName:}" failed. No retries permitted until 2026-07-21 17:26:54.659986032 +0000 UTC m=+66.231607767 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs") pod "network-metrics-daemon-9r9z4" (UID: "cb83c0b0-6020-43bf-be65-b41d51b76e5a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:22.660176 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:22.660022    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret podName:e5718cfc-649b-4cd1-bd58-0ad5494c5e56 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:54.660014099 +0000 UTC m=+66.231635832 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret") pod "global-pull-secret-syncer-qht4v" (UID: "e5718cfc-649b-4cd1-bd58-0ad5494c5e56") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:22.916658 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:22.916625    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:22.917140 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:22.916756    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qht4v" podUID="e5718cfc-649b-4cd1-bd58-0ad5494c5e56"
Jul 21 17:26:23.807565 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.807537    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-228.ec2.internal" event="NodeReady"
Jul 21 17:26:23.807745 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.807695    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 21 17:26:23.861897 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.861867    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8zf7t"]
Jul 21 17:26:23.863989 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.863967    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r9btt"]
Jul 21 17:26:23.864143 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.864114    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:23.865547 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.865526    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r9btt"
Jul 21 17:26:23.868662 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.868216    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 21 17:26:23.868662 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.868363    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 21 17:26:23.868662 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.868423    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 21 17:26:23.868662 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.868449    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 21 17:26:23.868662 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.868476    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d5qz9\""
Jul 21 17:26:23.868662 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.868583    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 21 17:26:23.869780 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.869763    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c7lxh\""
Jul 21 17:26:23.875094 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.875051    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r9btt"]
Jul 21 17:26:23.878272 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.878249    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8zf7t"]
Jul 21 17:26:23.916548 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.916520    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:26:23.916731 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.916557    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:23.919394 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.919373    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 21 17:26:23.919394 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.919393    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 21 17:26:23.919577 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.919469    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-swxfv\""
Jul 21 17:26:23.919649 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.919632    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 21 17:26:23.919809 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.919780    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gndbp\""
Jul 21 17:26:23.971187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.971148    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/215830f3-8192-4423-a3fa-269ea3625142-cert\") pod \"ingress-canary-r9btt\" (UID: \"215830f3-8192-4423-a3fa-269ea3625142\") " pod="openshift-ingress-canary/ingress-canary-r9btt"
Jul 21 17:26:23.971350 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.971194    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-tmp-dir\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:23.971350 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.971228    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-config-volume\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:23.971350 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.971294    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdlvb\" (UniqueName: \"kubernetes.io/projected/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-kube-api-access-hdlvb\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:23.971519 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.971381    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:23.971519 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:23.971419    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps89c\" (UniqueName: \"kubernetes.io/projected/215830f3-8192-4423-a3fa-269ea3625142-kube-api-access-ps89c\") pod \"ingress-canary-r9btt\" (UID: \"215830f3-8192-4423-a3fa-269ea3625142\") " pod="openshift-ingress-canary/ingress-canary-r9btt"
Jul 21 17:26:24.072449 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:24.072362    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/215830f3-8192-4423-a3fa-269ea3625142-cert\") pod \"ingress-canary-r9btt\" (UID: \"215830f3-8192-4423-a3fa-269ea3625142\") " pod="openshift-ingress-canary/ingress-canary-r9btt"
Jul 21 17:26:24.072449 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:24.072401    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-tmp-dir\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:24.072449 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:24.072441    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-config-volume\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:24.072707 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:24.072461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdlvb\" (UniqueName: \"kubernetes.io/projected/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-kube-api-access-hdlvb\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:24.072707 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:24.072645    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:24.072707 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:24.072682    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps89c\" (UniqueName: \"kubernetes.io/projected/215830f3-8192-4423-a3fa-269ea3625142-kube-api-access-ps89c\") pod \"ingress-canary-r9btt\" (UID: \"215830f3-8192-4423-a3fa-269ea3625142\") " pod="openshift-ingress-canary/ingress-canary-r9btt"
Jul 21 17:26:24.072834 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:24.072794    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-tmp-dir\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:24.072882 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:24.072854    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:24.072925 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:24.072903    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls podName:6ee3e5a6-186c-46a2-aef3-362ea08f9b30 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:24.572888392 +0000 UTC m=+36.144510124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls") pod "dns-default-8zf7t" (UID: "6ee3e5a6-186c-46a2-aef3-362ea08f9b30") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:24.073250 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:24.073230    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-config-volume\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:24.077250 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:24.077227    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/215830f3-8192-4423-a3fa-269ea3625142-cert\") pod \"ingress-canary-r9btt\" (UID: \"215830f3-8192-4423-a3fa-269ea3625142\") " pod="openshift-ingress-canary/ingress-canary-r9btt"
Jul 21 17:26:24.084523 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:24.084483    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdlvb\" (UniqueName: \"kubernetes.io/projected/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-kube-api-access-hdlvb\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:24.084820 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:24.084798    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps89c\" (UniqueName: \"kubernetes.io/projected/215830f3-8192-4423-a3fa-269ea3625142-kube-api-access-ps89c\") pod \"ingress-canary-r9btt\" (UID: \"215830f3-8192-4423-a3fa-269ea3625142\") " pod="openshift-ingress-canary/ingress-canary-r9btt"
Jul 21 17:26:24.183334 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:24.183299    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r9btt"
Jul 21 17:26:24.576868 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:24.576832    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:24.577028 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:24.577001    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:24.577126 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:24.577109    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls podName:6ee3e5a6-186c-46a2-aef3-362ea08f9b30 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:25.577088822 +0000 UTC m=+37.148710555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls") pod "dns-default-8zf7t" (UID: "6ee3e5a6-186c-46a2-aef3-362ea08f9b30") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:24.916707 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:24.916672    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:24.919838 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:24.919820    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 21 17:26:25.078868 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:25.078643    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r9btt"]
Jul 21 17:26:25.083258 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:26:25.083215    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215830f3_8192_4423_a3fa_269ea3625142.slice/crio-3bc5cc417f440442a6855dd686e18a08236d7fc135bd16c7b3a19a4538cd60ca WatchSource:0}: Error finding container 3bc5cc417f440442a6855dd686e18a08236d7fc135bd16c7b3a19a4538cd60ca: Status 404 returned error can't find the container with id 3bc5cc417f440442a6855dd686e18a08236d7fc135bd16c7b3a19a4538cd60ca
Jul 21 17:26:25.092320 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:25.092295    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rnxt" event={"ID":"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1","Type":"ContainerStarted","Data":"ec81c45eb0e855f9d896acd994c08dd95f791b4843214a8f95c9c6365313b1b1"}
Jul 21 17:26:25.093432 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:25.093409    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r9btt" event={"ID":"215830f3-8192-4423-a3fa-269ea3625142","Type":"ContainerStarted","Data":"3bc5cc417f440442a6855dd686e18a08236d7fc135bd16c7b3a19a4538cd60ca"}
Jul 21 17:26:25.584167 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:25.584056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:25.584352 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:25.584216    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:25.584352 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:25.584292    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls podName:6ee3e5a6-186c-46a2-aef3-362ea08f9b30 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:27.584269657 +0000 UTC m=+39.155891391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls") pod "dns-default-8zf7t" (UID: "6ee3e5a6-186c-46a2-aef3-362ea08f9b30") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:26.099485 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:26.099450    2562 generic.go:356] "Generic (PLEG): container finished" podID="7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1" containerID="ec81c45eb0e855f9d896acd994c08dd95f791b4843214a8f95c9c6365313b1b1" exitCode=0
Jul 21 17:26:26.100085 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:26.099503    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rnxt" event={"ID":"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1","Type":"ContainerDied","Data":"ec81c45eb0e855f9d896acd994c08dd95f791b4843214a8f95c9c6365313b1b1"}
Jul 21 17:26:27.105404 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:27.105292    2562 generic.go:356] "Generic (PLEG): container finished" podID="7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1" containerID="b7147adbc5f0ab7b50886c31f485dd43e484fd151b21d7fdc9a4963c25e2aa63" exitCode=0
Jul 21 17:26:27.105404 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:27.105353    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rnxt" event={"ID":"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1","Type":"ContainerDied","Data":"b7147adbc5f0ab7b50886c31f485dd43e484fd151b21d7fdc9a4963c25e2aa63"}
Jul 21 17:26:27.600236 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:27.600197    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:27.600402 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:27.600308    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:27.600402 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:27.600363    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls podName:6ee3e5a6-186c-46a2-aef3-362ea08f9b30 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:31.60034865 +0000 UTC m=+43.171970383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls") pod "dns-default-8zf7t" (UID: "6ee3e5a6-186c-46a2-aef3-362ea08f9b30") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:28.111036 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:28.110807    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rnxt" event={"ID":"7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1","Type":"ContainerStarted","Data":"482ce3c437574cdae2297c371e437708bd9fa50e22b98abd4bb2dff5746eb3cc"}
Jul 21 17:26:28.112209 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:28.112187    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r9btt" event={"ID":"215830f3-8192-4423-a3fa-269ea3625142","Type":"ContainerStarted","Data":"e9e4bd592b7c638b2cc084284f2283a9a93ec7ec9bb1d76b9d66d76fc5869568"}
Jul 21 17:26:28.137245 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:28.137174    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7rnxt" podStartSLOduration=10.028357049 podStartE2EDuration="39.137157388s" totalImagesPullingTime="29.108800339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:28.136889552 +0000 UTC m=+39.708511308" watchObservedRunningTime="2026-07-21 17:26:28.137157388 +0000 UTC m=+39.708779143"
Jul 21 17:26:28.154043 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:28.153993    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r9btt" podStartSLOduration=2.5938881 podStartE2EDuration="5.153977456s" totalImagesPullingTime="2.560089356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:28.153608924 +0000 UTC m=+39.725230680" watchObservedRunningTime="2026-07-21 17:26:28.153977456 +0000 UTC m=+39.725599210"
Jul 21 17:26:31.631822 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:31.631780    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:31.632365 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:31.631922    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:31.632365 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:31.632001    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls podName:6ee3e5a6-186c-46a2-aef3-362ea08f9b30 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:39.631976799 +0000 UTC m=+51.203598533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls") pod "dns-default-8zf7t" (UID: "6ee3e5a6-186c-46a2-aef3-362ea08f9b30") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:39.692152 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:39.692110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:39.692758 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:39.692285    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:39.692758 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:39.692436    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls podName:6ee3e5a6-186c-46a2-aef3-362ea08f9b30 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:55.692403444 +0000 UTC m=+67.264025190 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls") pod "dns-default-8zf7t" (UID: "6ee3e5a6-186c-46a2-aef3-362ea08f9b30") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:50.097935 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:50.097906    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdplm"
Jul 21 17:26:54.600767 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:54.600726    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wndt4\" (UniqueName: \"kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4\") pod \"network-check-target-r6qp2\" (UID: \"c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4\") " pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:54.603949 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:54.603926    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 21 17:26:54.614506 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:54.614476    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 21 17:26:54.626014 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:54.625982    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wndt4\" (UniqueName: \"kubernetes.io/projected/c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4-kube-api-access-wndt4\") pod \"network-check-target-r6qp2\" (UID: \"c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4\") " pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:54.701326 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:54.701291    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret\") pod \"global-pull-secret-syncer-qht4v\" (UID: \"e5718cfc-649b-4cd1-bd58-0ad5494c5e56\") " pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:54.701490 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:54.701345    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs\") pod \"network-metrics-daemon-9r9z4\" (UID: \"cb83c0b0-6020-43bf-be65-b41d51b76e5a\") " pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:26:54.704226 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:54.704203    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 21 17:26:54.704355 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:54.704334    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 21 17:26:54.711691 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:54.711660    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 21 17:26:54.711814 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:54.711745    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs podName:cb83c0b0-6020-43bf-be65-b41d51b76e5a nodeName:}" failed. No retries permitted until 2026-07-21 17:27:58.711727178 +0000 UTC m=+130.283348910 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs") pod "network-metrics-daemon-9r9z4" (UID: "cb83c0b0-6020-43bf-be65-b41d51b76e5a") : secret "metrics-daemon-secret" not found
Jul 21 17:26:54.713893 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:54.713867    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5718cfc-649b-4cd1-bd58-0ad5494c5e56-original-pull-secret\") pod \"global-pull-secret-syncer-qht4v\" (UID: \"e5718cfc-649b-4cd1-bd58-0ad5494c5e56\") " pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:54.837661 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:54.837631    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gndbp\""
Jul 21 17:26:54.844857 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:54.844835    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:26:54.927322 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:54.927296    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qht4v"
Jul 21 17:26:54.972563 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:54.972529    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r6qp2"]
Jul 21 17:26:54.977753 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:26:54.977714    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12e1bdf_b386_4a13_b3aa_a2a8b0ea4ab4.slice/crio-0770196e2845b8e04c84f9cac2423a28ae9cce280199aebbf274165274c41786 WatchSource:0}: Error finding container 0770196e2845b8e04c84f9cac2423a28ae9cce280199aebbf274165274c41786: Status 404 returned error can't find the container with id 0770196e2845b8e04c84f9cac2423a28ae9cce280199aebbf274165274c41786
Jul 21 17:26:55.054913 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:55.054841    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qht4v"]
Jul 21 17:26:55.057312 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:26:55.057286    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5718cfc_649b_4cd1_bd58_0ad5494c5e56.slice/crio-f8734feadca400083056ee4812e61372cc6edc6f4a7c9c6425d212621019f109 WatchSource:0}: Error finding container f8734feadca400083056ee4812e61372cc6edc6f4a7c9c6425d212621019f109: Status 404 returned error can't find the container with id f8734feadca400083056ee4812e61372cc6edc6f4a7c9c6425d212621019f109
Jul 21 17:26:55.174742 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:55.174710    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qht4v" event={"ID":"e5718cfc-649b-4cd1-bd58-0ad5494c5e56","Type":"ContainerStarted","Data":"f8734feadca400083056ee4812e61372cc6edc6f4a7c9c6425d212621019f109"}
Jul 21 17:26:55.175793 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:55.175767    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r6qp2" event={"ID":"c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4","Type":"ContainerStarted","Data":"0770196e2845b8e04c84f9cac2423a28ae9cce280199aebbf274165274c41786"}
Jul 21 17:26:55.710673 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:26:55.710634    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:26:55.711130 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:55.710796    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:55.711130 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:26:55.710863    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls podName:6ee3e5a6-186c-46a2-aef3-362ea08f9b30 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:27.710843907 +0000 UTC m=+99.282465642 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls") pod "dns-default-8zf7t" (UID: "6ee3e5a6-186c-46a2-aef3-362ea08f9b30") : secret "dns-default-metrics-tls" not found
Jul 21 17:27:00.191383 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:00.191339    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qht4v" event={"ID":"e5718cfc-649b-4cd1-bd58-0ad5494c5e56","Type":"ContainerStarted","Data":"ab9d35772b49b155a60917da90a03876ebc68adc8048fbf1ce34c28589753d74"}
Jul 21 17:27:00.192665 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:00.192640    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r6qp2" event={"ID":"c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4","Type":"ContainerStarted","Data":"a0b2f9cb466385018118d92d566663bc08144474b04957bf44d63164df942881"}
Jul 21 17:27:00.192774 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:00.192715    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:27:00.207977 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:00.207926    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qht4v" podStartSLOduration=66.790495349 podStartE2EDuration="1m11.207912308s" totalImagesPullingTime="4.417416959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:00.207161367 +0000 UTC m=+71.778783123" watchObservedRunningTime="2026-07-21 17:27:00.207912308 +0000 UTC m=+71.779534063"
Jul 21 17:27:00.227123 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:00.227048    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r6qp2" podStartSLOduration=67.734032663 podStartE2EDuration="1m12.227030878s" totalImagesPullingTime="4.492998215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:00.226215729 +0000 UTC m=+71.797837484" watchObservedRunningTime="2026-07-21 17:27:00.227030878 +0000 UTC m=+71.798652632"
Jul 21 17:27:02.600567 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.600536    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-9qvnl"]
Jul 21 17:27:02.604572 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.604545    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.609082 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.609038    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 21 17:27:02.609213 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.609059    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 21 17:27:02.609213 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.609204    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vdcsq\""
Jul 21 17:27:02.609900 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.609871    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 21 17:27:02.610810 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.610693    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 21 17:27:02.614646 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.614627    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-9qvnl"]
Jul 21 17:27:02.618145 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.618127    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 21 17:27:02.662915 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.662890    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a37a390-191f-4644-b398-6dcf97268ee8-service-ca-bundle\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.663045 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.662922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a37a390-191f-4644-b398-6dcf97268ee8-tmp\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.663045 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.662939    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a37a390-191f-4644-b398-6dcf97268ee8-serving-cert\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.663045 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.663004    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a37a390-191f-4644-b398-6dcf97268ee8-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.663170 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.663118    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0a37a390-191f-4644-b398-6dcf97268ee8-snapshots\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.663170 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.663140    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc2fs\" (UniqueName: \"kubernetes.io/projected/0a37a390-191f-4644-b398-6dcf97268ee8-kube-api-access-fc2fs\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.764418 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.764368    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a37a390-191f-4644-b398-6dcf97268ee8-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.764598 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.764458    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0a37a390-191f-4644-b398-6dcf97268ee8-snapshots\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.764598 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.764518    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc2fs\" (UniqueName: \"kubernetes.io/projected/0a37a390-191f-4644-b398-6dcf97268ee8-kube-api-access-fc2fs\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.764598 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.764558    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a37a390-191f-4644-b398-6dcf97268ee8-service-ca-bundle\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.764598 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.764593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a37a390-191f-4644-b398-6dcf97268ee8-tmp\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.764783 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.764609    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a37a390-191f-4644-b398-6dcf97268ee8-serving-cert\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.765098 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.765052    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a37a390-191f-4644-b398-6dcf97268ee8-tmp\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.765177 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.765116    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0a37a390-191f-4644-b398-6dcf97268ee8-snapshots\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.765216 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.765183    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a37a390-191f-4644-b398-6dcf97268ee8-service-ca-bundle\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.765432 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.765417    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a37a390-191f-4644-b398-6dcf97268ee8-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.767007 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.766989    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a37a390-191f-4644-b398-6dcf97268ee8-serving-cert\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.773510 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.773490    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc2fs\" (UniqueName: \"kubernetes.io/projected/0a37a390-191f-4644-b398-6dcf97268ee8-kube-api-access-fc2fs\") pod \"insights-operator-555d9d7885-9qvnl\" (UID: \"0a37a390-191f-4644-b398-6dcf97268ee8\") " pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:02.913618 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:02.913589    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-9qvnl"
Jul 21 17:27:03.049865 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:03.049833    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-9qvnl"]
Jul 21 17:27:03.053589 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:27:03.053565    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a37a390_191f_4644_b398_6dcf97268ee8.slice/crio-eb94c327cbfe859b1bfb7cc7c9588241caf457fe69cb150e4896bbcad9431351 WatchSource:0}: Error finding container eb94c327cbfe859b1bfb7cc7c9588241caf457fe69cb150e4896bbcad9431351: Status 404 returned error can't find the container with id eb94c327cbfe859b1bfb7cc7c9588241caf457fe69cb150e4896bbcad9431351
Jul 21 17:27:03.200655 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:03.200569    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-9qvnl" event={"ID":"0a37a390-191f-4644-b398-6dcf97268ee8","Type":"ContainerStarted","Data":"eb94c327cbfe859b1bfb7cc7c9588241caf457fe69cb150e4896bbcad9431351"}
Jul 21 17:27:07.214999 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.214944    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-9qvnl" event={"ID":"0a37a390-191f-4644-b398-6dcf97268ee8","Type":"ContainerStarted","Data":"85e1f309d706aa2ca83e628669ba5503a0b83ce189eeed309798d5772db077da"}
Jul 21 17:27:07.244704 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.244650    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-9qvnl" podStartSLOduration=1.5672872 podStartE2EDuration="5.244629603s" totalImagesPullingTime="3.677342403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:07.244215627 +0000 UTC m=+78.815837403" watchObservedRunningTime="2026-07-21 17:27:07.244629603 +0000 UTC m=+78.816251362"
Jul 21 17:27:07.629810 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.629730    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6btrk"]
Jul 21 17:27:07.632654 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.632634    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:07.635179 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.635158    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 21 17:27:07.635275 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.635176    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pmcsw\""
Jul 21 17:27:07.636254 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.636237    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 21 17:27:07.656057 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.656031    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6btrk"]
Jul 21 17:27:07.700205 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.700170    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:07.700369 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.700222    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:07.700369 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.700337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-crio-socket\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:07.700439 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.700413    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6tf6\" (UniqueName: \"kubernetes.io/projected/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-api-access-j6tf6\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:07.700481 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.700443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-data-volume\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:07.800877 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.800839    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-crio-socket\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:07.801088 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.800888    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6tf6\" (UniqueName: \"kubernetes.io/projected/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-api-access-j6tf6\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:07.801088 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.800913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-data-volume\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:07.801088 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.800949    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:07.801088 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.800977    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-crio-socket\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:07.801088 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.800993    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:07.801347 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:07.801088    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 17:27:07.801347 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:07.801132    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:07.801347 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:07.801174    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm podName:5757dbc1-e93d-4fcd-b888-c3e7cb4955b2 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:08.301152852 +0000 UTC m=+79.872774614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6btrk" (UID: "5757dbc1-e93d-4fcd-b888-c3e7cb4955b2") : configmap "kube-rbac-proxy" not found
Jul 21 17:27:07.801347 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:07.801200    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls podName:5757dbc1-e93d-4fcd-b888-c3e7cb4955b2 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:08.301184359 +0000 UTC m=+79.872806107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6btrk" (UID: "5757dbc1-e93d-4fcd-b888-c3e7cb4955b2") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:07.801347 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.801311    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-data-volume\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:07.816972 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:07.816940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6tf6\" (UniqueName: \"kubernetes.io/projected/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-api-access-j6tf6\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:08.304779 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:08.304738    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:08.305396 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:08.304799    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:08.305396 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:08.304868    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 17:27:08.305396 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:08.304941    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm podName:5757dbc1-e93d-4fcd-b888-c3e7cb4955b2 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:09.304921296 +0000 UTC m=+80.876543031 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6btrk" (UID: "5757dbc1-e93d-4fcd-b888-c3e7cb4955b2") : configmap "kube-rbac-proxy" not found
Jul 21 17:27:08.305396 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:08.305010    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:08.305396 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:08.305093    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls podName:5757dbc1-e93d-4fcd-b888-c3e7cb4955b2 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:09.305049906 +0000 UTC m=+80.876671655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6btrk" (UID: "5757dbc1-e93d-4fcd-b888-c3e7cb4955b2") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:09.313264 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.313149    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:09.313264 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.313192    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:09.313676 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:09.313273    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 17:27:09.313676 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:09.313288    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:09.313676 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:09.313338    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm podName:5757dbc1-e93d-4fcd-b888-c3e7cb4955b2 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:11.31332128 +0000 UTC m=+82.884943013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6btrk" (UID: "5757dbc1-e93d-4fcd-b888-c3e7cb4955b2") : configmap "kube-rbac-proxy" not found
Jul 21 17:27:09.313676 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:09.313351    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls podName:5757dbc1-e93d-4fcd-b888-c3e7cb4955b2 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:11.313345146 +0000 UTC m=+82.884966878 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6btrk" (UID: "5757dbc1-e93d-4fcd-b888-c3e7cb4955b2") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:09.500081 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.500039    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8hdhq"]
Jul 21 17:27:09.502750 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.502734    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8hdhq"
Jul 21 17:27:09.505481 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.505450    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 21 17:27:09.505481 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.505456    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 21 17:27:09.505645 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.505495    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fh7bc\""
Jul 21 17:27:09.514574 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.514552    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8hdhq"]
Jul 21 17:27:09.602105 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.602012    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8gvpd_85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15/dns-node-resolver/0.log"
Jul 21 17:27:09.610799 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.610773    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2"]
Jul 21 17:27:09.613658 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.613644    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2"
Jul 21 17:27:09.616443 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.616239    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lnbg\" (UniqueName: \"kubernetes.io/projected/52802b89-ae19-49fa-bc8c-6bb23a60b455-kube-api-access-9lnbg\") pod \"volume-data-source-validator-78cf4c6f6-8hdhq\" (UID: \"52802b89-ae19-49fa-bc8c-6bb23a60b455\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8hdhq"
Jul 21 17:27:09.619543 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.619523    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 21 17:27:09.619676 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.619569    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 21 17:27:09.619777 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.619764    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 21 17:27:09.620376 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.620356    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wzpx6\""
Jul 21 17:27:09.644447 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.644410    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2"]
Jul 21 17:27:09.717334 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.717294    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-48bq2\" (UID: \"d95df149-45c7-430b-b86b-b8101475b4ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2"
Jul 21 17:27:09.717478 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.717362    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lnbg\" (UniqueName: \"kubernetes.io/projected/52802b89-ae19-49fa-bc8c-6bb23a60b455-kube-api-access-9lnbg\") pod \"volume-data-source-validator-78cf4c6f6-8hdhq\" (UID: \"52802b89-ae19-49fa-bc8c-6bb23a60b455\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8hdhq"
Jul 21 17:27:09.717478 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.717383    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6rfm\" (UniqueName: \"kubernetes.io/projected/d95df149-45c7-430b-b86b-b8101475b4ad-kube-api-access-n6rfm\") pod \"cluster-samples-operator-79b7c55f87-48bq2\" (UID: \"d95df149-45c7-430b-b86b-b8101475b4ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2"
Jul 21 17:27:09.727682 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.727650    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lnbg\" (UniqueName: \"kubernetes.io/projected/52802b89-ae19-49fa-bc8c-6bb23a60b455-kube-api-access-9lnbg\") pod \"volume-data-source-validator-78cf4c6f6-8hdhq\" (UID: \"52802b89-ae19-49fa-bc8c-6bb23a60b455\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8hdhq"
Jul 21 17:27:09.811545 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.811513    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8hdhq"
Jul 21 17:27:09.818632 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.818609    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6rfm\" (UniqueName: \"kubernetes.io/projected/d95df149-45c7-430b-b86b-b8101475b4ad-kube-api-access-n6rfm\") pod \"cluster-samples-operator-79b7c55f87-48bq2\" (UID: \"d95df149-45c7-430b-b86b-b8101475b4ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2"
Jul 21 17:27:09.818750 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.818667    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-48bq2\" (UID: \"d95df149-45c7-430b-b86b-b8101475b4ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2"
Jul 21 17:27:09.818812 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:09.818761    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 21 17:27:09.818865 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:09.818813    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls podName:d95df149-45c7-430b-b86b-b8101475b4ad nodeName:}" failed. No retries permitted until 2026-07-21 17:27:10.318797645 +0000 UTC m=+81.890419377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-48bq2" (UID: "d95df149-45c7-430b-b86b-b8101475b4ad") : secret "samples-operator-tls" not found
Jul 21 17:27:09.829085 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.829043    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6rfm\" (UniqueName: \"kubernetes.io/projected/d95df149-45c7-430b-b86b-b8101475b4ad-kube-api-access-n6rfm\") pod \"cluster-samples-operator-79b7c55f87-48bq2\" (UID: \"d95df149-45c7-430b-b86b-b8101475b4ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2"
Jul 21 17:27:09.940305 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:09.940278    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8hdhq"]
Jul 21 17:27:09.944108 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:27:09.944083    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52802b89_ae19_49fa_bc8c_6bb23a60b455.slice/crio-55884edb453e940bbbd3bcff529ab9df1989924ed0511d26df1a0bd6d9a4e493 WatchSource:0}: Error finding container 55884edb453e940bbbd3bcff529ab9df1989924ed0511d26df1a0bd6d9a4e493: Status 404 returned error can't find the container with id 55884edb453e940bbbd3bcff529ab9df1989924ed0511d26df1a0bd6d9a4e493
Jul 21 17:27:10.224016 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.223979    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8hdhq" event={"ID":"52802b89-ae19-49fa-bc8c-6bb23a60b455","Type":"ContainerStarted","Data":"55884edb453e940bbbd3bcff529ab9df1989924ed0511d26df1a0bd6d9a4e493"}
Jul 21 17:27:10.322439 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.322404    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-48bq2\" (UID: \"d95df149-45c7-430b-b86b-b8101475b4ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2"
Jul 21 17:27:10.322797 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:10.322525    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 21 17:27:10.322797 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:10.322578    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls podName:d95df149-45c7-430b-b86b-b8101475b4ad nodeName:}" failed. No retries permitted until 2026-07-21 17:27:11.322563663 +0000 UTC m=+82.894185396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-48bq2" (UID: "d95df149-45c7-430b-b86b-b8101475b4ad") : secret "samples-operator-tls" not found
Jul 21 17:27:10.508645 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.508563    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-7vppx"]
Jul 21 17:27:10.511783 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.511760    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:10.514662 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.514637    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 21 17:27:10.514793 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.514682    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 21 17:27:10.514868 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.514852    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 21 17:27:10.515207 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.514927    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 21 17:27:10.515207 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.515037    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-mkn49\""
Jul 21 17:27:10.520595 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.520566    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 21 17:27:10.527443 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.527422    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-7vppx"]
Jul 21 17:27:10.596197 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.596171    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xxbb8_b35c0345-124e-40c5-8de6-2b9c3122e24a/node-ca/0.log"
Jul 21 17:27:10.624549 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.624505    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtnhb\" (UniqueName: \"kubernetes.io/projected/2302a4fe-a2b9-4de0-8dce-2805b3c2d825-kube-api-access-rtnhb\") pod \"console-operator-565b7b569d-7vppx\" (UID: \"2302a4fe-a2b9-4de0-8dce-2805b3c2d825\") " pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:10.624718 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.624576    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2302a4fe-a2b9-4de0-8dce-2805b3c2d825-serving-cert\") pod \"console-operator-565b7b569d-7vppx\" (UID: \"2302a4fe-a2b9-4de0-8dce-2805b3c2d825\") " pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:10.624718 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.624653    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2302a4fe-a2b9-4de0-8dce-2805b3c2d825-trusted-ca\") pod \"console-operator-565b7b569d-7vppx\" (UID: \"2302a4fe-a2b9-4de0-8dce-2805b3c2d825\") " pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:10.624718 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.624689    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2302a4fe-a2b9-4de0-8dce-2805b3c2d825-config\") pod \"console-operator-565b7b569d-7vppx\" (UID: \"2302a4fe-a2b9-4de0-8dce-2805b3c2d825\") " pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:10.725756 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.725714    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2302a4fe-a2b9-4de0-8dce-2805b3c2d825-serving-cert\") pod \"console-operator-565b7b569d-7vppx\" (UID: \"2302a4fe-a2b9-4de0-8dce-2805b3c2d825\") " pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:10.725943 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.725773    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2302a4fe-a2b9-4de0-8dce-2805b3c2d825-trusted-ca\") pod \"console-operator-565b7b569d-7vppx\" (UID: \"2302a4fe-a2b9-4de0-8dce-2805b3c2d825\") " pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:10.725943 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.725798    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2302a4fe-a2b9-4de0-8dce-2805b3c2d825-config\") pod \"console-operator-565b7b569d-7vppx\" (UID: \"2302a4fe-a2b9-4de0-8dce-2805b3c2d825\") " pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:10.725943 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.725854    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtnhb\" (UniqueName: \"kubernetes.io/projected/2302a4fe-a2b9-4de0-8dce-2805b3c2d825-kube-api-access-rtnhb\") pod \"console-operator-565b7b569d-7vppx\" (UID: \"2302a4fe-a2b9-4de0-8dce-2805b3c2d825\") " pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:10.726719 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.726690    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2302a4fe-a2b9-4de0-8dce-2805b3c2d825-config\") pod \"console-operator-565b7b569d-7vppx\" (UID: \"2302a4fe-a2b9-4de0-8dce-2805b3c2d825\") " pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:10.726855 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.726789    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2302a4fe-a2b9-4de0-8dce-2805b3c2d825-trusted-ca\") pod \"console-operator-565b7b569d-7vppx\" (UID: \"2302a4fe-a2b9-4de0-8dce-2805b3c2d825\") " pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:10.728842 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.728809    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2302a4fe-a2b9-4de0-8dce-2805b3c2d825-serving-cert\") pod \"console-operator-565b7b569d-7vppx\" (UID: \"2302a4fe-a2b9-4de0-8dce-2805b3c2d825\") " pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:10.736041 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.736020    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtnhb\" (UniqueName: \"kubernetes.io/projected/2302a4fe-a2b9-4de0-8dce-2805b3c2d825-kube-api-access-rtnhb\") pod \"console-operator-565b7b569d-7vppx\" (UID: \"2302a4fe-a2b9-4de0-8dce-2805b3c2d825\") " pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:10.823107 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:10.822995    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:11.078375 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:11.078351    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-7vppx"]
Jul 21 17:27:11.080451 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:27:11.080425    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2302a4fe_a2b9_4de0_8dce_2805b3c2d825.slice/crio-7703a0a809ecdc8289b65e947220574f4fd5b64597844beb820a494b323fe9e8 WatchSource:0}: Error finding container 7703a0a809ecdc8289b65e947220574f4fd5b64597844beb820a494b323fe9e8: Status 404 returned error can't find the container with id 7703a0a809ecdc8289b65e947220574f4fd5b64597844beb820a494b323fe9e8
Jul 21 17:27:11.227775 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:11.227739    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-7vppx" event={"ID":"2302a4fe-a2b9-4de0-8dce-2805b3c2d825","Type":"ContainerStarted","Data":"7703a0a809ecdc8289b65e947220574f4fd5b64597844beb820a494b323fe9e8"}
Jul 21 17:27:11.229046 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:11.229024    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8hdhq" event={"ID":"52802b89-ae19-49fa-bc8c-6bb23a60b455","Type":"ContainerStarted","Data":"00836c13a081e8fdbf8c2dd269e1ed45e77b9c8a10dbd664863b3346374d3cff"}
Jul 21 17:27:11.247585 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:11.247541    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8hdhq" podStartSLOduration=1.194735983 podStartE2EDuration="2.247527356s" totalImagesPullingTime="1.052791373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:11.246574973 +0000 UTC m=+82.818196728" watchObservedRunningTime="2026-07-21 17:27:11.247527356 +0000 UTC m=+82.819149110"
Jul 21 17:27:11.329981 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:11.329894    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-48bq2\" (UID: \"d95df149-45c7-430b-b86b-b8101475b4ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2"
Jul 21 17:27:11.329981 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:11.329943    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:11.330474 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:11.329985    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:11.330474 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:11.330036    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 21 17:27:11.330474 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:11.330058    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 17:27:11.330474 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:11.330110    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:11.330474 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:11.330115    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls podName:d95df149-45c7-430b-b86b-b8101475b4ad nodeName:}" failed. No retries permitted until 2026-07-21 17:27:13.330099207 +0000 UTC m=+84.901720940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-48bq2" (UID: "d95df149-45c7-430b-b86b-b8101475b4ad") : secret "samples-operator-tls" not found
Jul 21 17:27:11.330474 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:11.330163    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm podName:5757dbc1-e93d-4fcd-b888-c3e7cb4955b2 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:15.330146484 +0000 UTC m=+86.901768218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6btrk" (UID: "5757dbc1-e93d-4fcd-b888-c3e7cb4955b2") : configmap "kube-rbac-proxy" not found
Jul 21 17:27:11.330474 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:11.330178    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls podName:5757dbc1-e93d-4fcd-b888-c3e7cb4955b2 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:15.330169435 +0000 UTC m=+86.901791172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6btrk" (UID: "5757dbc1-e93d-4fcd-b888-c3e7cb4955b2") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:11.396501 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:11.396466    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r9btt_215830f3-8192-4423-a3fa-269ea3625142/serve-healthcheck-canary/0.log"
Jul 21 17:27:12.500428 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.500393    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n"]
Jul 21 17:27:12.503129 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.503112    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n"
Jul 21 17:27:12.506281 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.506250    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 21 17:27:12.507148 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.507125    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-klpxq\""
Jul 21 17:27:12.507255 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.507130    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 21 17:27:12.507255 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.507224    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 21 17:27:12.507639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.507620    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 21 17:27:12.515343 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.515315    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n"]
Jul 21 17:27:12.641742 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.641709    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mvmm\" (UniqueName: \"kubernetes.io/projected/813ad497-8605-4f33-b614-32c185d701fc-kube-api-access-5mvmm\") pod \"kube-storage-version-migrator-operator-7658d9b749-56c8n\" (UID: \"813ad497-8605-4f33-b614-32c185d701fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n"
Jul 21 17:27:12.641932 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.641786    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/813ad497-8605-4f33-b614-32c185d701fc-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-56c8n\" (UID: \"813ad497-8605-4f33-b614-32c185d701fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n"
Jul 21 17:27:12.641932 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.641880    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/813ad497-8605-4f33-b614-32c185d701fc-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-56c8n\" (UID: \"813ad497-8605-4f33-b614-32c185d701fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n"
Jul 21 17:27:12.742331 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.742285    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/813ad497-8605-4f33-b614-32c185d701fc-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-56c8n\" (UID: \"813ad497-8605-4f33-b614-32c185d701fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n"
Jul 21 17:27:12.742525 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.742389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/813ad497-8605-4f33-b614-32c185d701fc-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-56c8n\" (UID: \"813ad497-8605-4f33-b614-32c185d701fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n"
Jul 21 17:27:12.742525 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.742428    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvmm\" (UniqueName: \"kubernetes.io/projected/813ad497-8605-4f33-b614-32c185d701fc-kube-api-access-5mvmm\") pod \"kube-storage-version-migrator-operator-7658d9b749-56c8n\" (UID: \"813ad497-8605-4f33-b614-32c185d701fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n"
Jul 21 17:27:12.743051 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.743020    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/813ad497-8605-4f33-b614-32c185d701fc-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-56c8n\" (UID: \"813ad497-8605-4f33-b614-32c185d701fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n"
Jul 21 17:27:12.744884 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.744861    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/813ad497-8605-4f33-b614-32c185d701fc-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-56c8n\" (UID: \"813ad497-8605-4f33-b614-32c185d701fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n"
Jul 21 17:27:12.752542 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.752473    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mvmm\" (UniqueName: \"kubernetes.io/projected/813ad497-8605-4f33-b614-32c185d701fc-kube-api-access-5mvmm\") pod \"kube-storage-version-migrator-operator-7658d9b749-56c8n\" (UID: \"813ad497-8605-4f33-b614-32c185d701fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n"
Jul 21 17:27:12.815939 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:12.815898    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n"
Jul 21 17:27:13.283899 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:13.283876    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n"]
Jul 21 17:27:13.287236 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:27:13.287196    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813ad497_8605_4f33_b614_32c185d701fc.slice/crio-df8bc0d8cdd4591885a24f7ad7dfef5df1be702dd84e1b3d22958ed361a56cb2 WatchSource:0}: Error finding container df8bc0d8cdd4591885a24f7ad7dfef5df1be702dd84e1b3d22958ed361a56cb2: Status 404 returned error can't find the container with id df8bc0d8cdd4591885a24f7ad7dfef5df1be702dd84e1b3d22958ed361a56cb2
Jul 21 17:27:13.348202 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:13.348165    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-48bq2\" (UID: \"d95df149-45c7-430b-b86b-b8101475b4ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2"
Jul 21 17:27:13.348352 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:13.348331    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 21 17:27:13.348426 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:13.348415    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls podName:d95df149-45c7-430b-b86b-b8101475b4ad nodeName:}" failed. No retries permitted until 2026-07-21 17:27:17.348396606 +0000 UTC m=+88.920018343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-48bq2" (UID: "d95df149-45c7-430b-b86b-b8101475b4ad") : secret "samples-operator-tls" not found
Jul 21 17:27:14.240492 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:14.240456    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/0.log"
Jul 21 17:27:14.240915 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:14.240505    2562 generic.go:356] "Generic (PLEG): container finished" podID="2302a4fe-a2b9-4de0-8dce-2805b3c2d825" containerID="a15704e6bbbd17f644dba34bfcbe072bac34db1ebb24cd766f56d7770aebd3e9" exitCode=255
Jul 21 17:27:14.240915 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:14.240592    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-7vppx" event={"ID":"2302a4fe-a2b9-4de0-8dce-2805b3c2d825","Type":"ContainerDied","Data":"a15704e6bbbd17f644dba34bfcbe072bac34db1ebb24cd766f56d7770aebd3e9"}
Jul 21 17:27:14.240915 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:14.240827    2562 scope.go:117] "RemoveContainer" containerID="a15704e6bbbd17f644dba34bfcbe072bac34db1ebb24cd766f56d7770aebd3e9"
Jul 21 17:27:14.241841 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:14.241815    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n" event={"ID":"813ad497-8605-4f33-b614-32c185d701fc","Type":"ContainerStarted","Data":"df8bc0d8cdd4591885a24f7ad7dfef5df1be702dd84e1b3d22958ed361a56cb2"}
Jul 21 17:27:15.249407 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:15.249377    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/1.log"
Jul 21 17:27:15.249822 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:15.249807    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/0.log"
Jul 21 17:27:15.249877 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:15.249842    2562 generic.go:356] "Generic (PLEG): container finished" podID="2302a4fe-a2b9-4de0-8dce-2805b3c2d825" containerID="31d61cd6c8306190dfeb09fd4a956172994c2c96b70b13f1aef83a69f1eb711f" exitCode=255
Jul 21 17:27:15.249928 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:15.249913    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-7vppx" event={"ID":"2302a4fe-a2b9-4de0-8dce-2805b3c2d825","Type":"ContainerDied","Data":"31d61cd6c8306190dfeb09fd4a956172994c2c96b70b13f1aef83a69f1eb711f"}
Jul 21 17:27:15.250543 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:15.250524    2562 scope.go:117] "RemoveContainer" containerID="a15704e6bbbd17f644dba34bfcbe072bac34db1ebb24cd766f56d7770aebd3e9"
Jul 21 17:27:15.250661 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:15.250623    2562 scope.go:117] "RemoveContainer" containerID="31d61cd6c8306190dfeb09fd4a956172994c2c96b70b13f1aef83a69f1eb711f"
Jul 21 17:27:15.250843 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:15.250823    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-7vppx_openshift-console-operator(2302a4fe-a2b9-4de0-8dce-2805b3c2d825)\"" pod="openshift-console-operator/console-operator-565b7b569d-7vppx" podUID="2302a4fe-a2b9-4de0-8dce-2805b3c2d825"
Jul 21 17:27:15.251840 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:15.251569    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n" event={"ID":"813ad497-8605-4f33-b614-32c185d701fc","Type":"ContainerStarted","Data":"a300b3d58a8dc042e9a5998d7d1f5d36b47c4b078e74db0a428fa9858614ec63"}
Jul 21 17:27:15.291150 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:15.291038    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n" podStartSLOduration=1.5464631180000001 podStartE2EDuration="3.291023366s" totalImagesPullingTime="1.744560248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:15.290247741 +0000 UTC m=+86.861869525" watchObservedRunningTime="2026-07-21 17:27:15.291023366 +0000 UTC m=+86.862645122"
Jul 21 17:27:15.365949 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:15.365900    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:15.365949 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:15.365957    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:15.366214 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:15.366095    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:15.366214 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:15.366159    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls podName:5757dbc1-e93d-4fcd-b888-c3e7cb4955b2 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:23.36614085 +0000 UTC m=+94.937762596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6btrk" (UID: "5757dbc1-e93d-4fcd-b888-c3e7cb4955b2") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:15.366332 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:15.366217    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 17:27:15.366332 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:15.366247    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm podName:5757dbc1-e93d-4fcd-b888-c3e7cb4955b2 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:23.366238212 +0000 UTC m=+94.937859948 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6btrk" (UID: "5757dbc1-e93d-4fcd-b888-c3e7cb4955b2") : configmap "kube-rbac-proxy" not found
Jul 21 17:27:16.256554 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:16.256525    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/1.log"
Jul 21 17:27:16.256927 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:16.256900    2562 scope.go:117] "RemoveContainer" containerID="31d61cd6c8306190dfeb09fd4a956172994c2c96b70b13f1aef83a69f1eb711f"
Jul 21 17:27:16.257128 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:16.257109    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-7vppx_openshift-console-operator(2302a4fe-a2b9-4de0-8dce-2805b3c2d825)\"" pod="openshift-console-operator/console-operator-565b7b569d-7vppx" podUID="2302a4fe-a2b9-4de0-8dce-2805b3c2d825"
Jul 21 17:27:17.379963 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:17.379917    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-48bq2\" (UID: \"d95df149-45c7-430b-b86b-b8101475b4ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2"
Jul 21 17:27:17.380381 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:17.380059    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 21 17:27:17.380381 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:17.380164    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls podName:d95df149-45c7-430b-b86b-b8101475b4ad nodeName:}" failed. No retries permitted until 2026-07-21 17:27:25.380141552 +0000 UTC m=+96.951763286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-48bq2" (UID: "d95df149-45c7-430b-b86b-b8101475b4ad") : secret "samples-operator-tls" not found
Jul 21 17:27:20.823687 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:20.823642    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:20.824214 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:20.823733    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:20.824214 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:20.824057    2562 scope.go:117] "RemoveContainer" containerID="31d61cd6c8306190dfeb09fd4a956172994c2c96b70b13f1aef83a69f1eb711f"
Jul 21 17:27:20.824292 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:20.824226    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-7vppx_openshift-console-operator(2302a4fe-a2b9-4de0-8dce-2805b3c2d825)\"" pod="openshift-console-operator/console-operator-565b7b569d-7vppx" podUID="2302a4fe-a2b9-4de0-8dce-2805b3c2d825"
Jul 21 17:27:23.425254 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:23.425192    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:23.425254 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:23.425259    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:23.425717 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:23.425383    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:23.425717 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:23.425435    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls podName:5757dbc1-e93d-4fcd-b888-c3e7cb4955b2 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:39.425420447 +0000 UTC m=+110.997042180 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6btrk" (UID: "5757dbc1-e93d-4fcd-b888-c3e7cb4955b2") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:23.425717 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:23.425693    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:25.441242 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:25.441202    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-48bq2\" (UID: \"d95df149-45c7-430b-b86b-b8101475b4ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2"
Jul 21 17:27:25.443836 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:25.443814    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d95df149-45c7-430b-b86b-b8101475b4ad-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-48bq2\" (UID: \"d95df149-45c7-430b-b86b-b8101475b4ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2"
Jul 21 17:27:25.525300 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:25.525266    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2"
Jul 21 17:27:25.651841 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:25.651816    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2"]
Jul 21 17:27:26.287948 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:26.287907    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2" event={"ID":"d95df149-45c7-430b-b86b-b8101475b4ad","Type":"ContainerStarted","Data":"cc885571c812c924cf9748b7d3b84e2dcaece992543a3007c5c80d1f8f3b2ffd"}
Jul 21 17:27:27.762962 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:27.762931    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:27:27.765336 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:27.765311    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ee3e5a6-186c-46a2-aef3-362ea08f9b30-metrics-tls\") pod \"dns-default-8zf7t\" (UID: \"6ee3e5a6-186c-46a2-aef3-362ea08f9b30\") " pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:27:27.779214 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:27.779190    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c7lxh\""
Jul 21 17:27:27.787231 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:27.787208    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:27:27.911384 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:27.911347    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8zf7t"]
Jul 21 17:27:27.914216 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:27:27.914192    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee3e5a6_186c_46a2_aef3_362ea08f9b30.slice/crio-5c50c06d3c58584267d189f187e02c2175b38355101f7d0794bc9d845ebf3ba8 WatchSource:0}: Error finding container 5c50c06d3c58584267d189f187e02c2175b38355101f7d0794bc9d845ebf3ba8: Status 404 returned error can't find the container with id 5c50c06d3c58584267d189f187e02c2175b38355101f7d0794bc9d845ebf3ba8
Jul 21 17:27:28.295506 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:28.295454    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8zf7t" event={"ID":"6ee3e5a6-186c-46a2-aef3-362ea08f9b30","Type":"ContainerStarted","Data":"5c50c06d3c58584267d189f187e02c2175b38355101f7d0794bc9d845ebf3ba8"}
Jul 21 17:27:28.297599 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:28.297569    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2" event={"ID":"d95df149-45c7-430b-b86b-b8101475b4ad","Type":"ContainerStarted","Data":"dfd573600c77cb8df565af04d8bc4227c229f3e7497cab9f9b6b7ff029959dcf"}
Jul 21 17:27:28.297733 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:28.297631    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2" event={"ID":"d95df149-45c7-430b-b86b-b8101475b4ad","Type":"ContainerStarted","Data":"23f1df6241aa8c3c9909e1d5050f8d885c7d9d08a32caa43e1790020239ec862"}
Jul 21 17:27:28.318384 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:28.318282    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-48bq2" podStartSLOduration=17.753668962 podStartE2EDuration="19.318259375s" totalImagesPullingTime="1.564590413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:28.317741673 +0000 UTC m=+99.889363428" watchObservedRunningTime="2026-07-21 17:27:28.318259375 +0000 UTC m=+99.889881133"
Jul 21 17:27:30.305918 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:30.305883    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8zf7t" event={"ID":"6ee3e5a6-186c-46a2-aef3-362ea08f9b30","Type":"ContainerStarted","Data":"638b0cca9886f86a5c6f6f7e79a99d0439163281132797b015c54812d5af289f"}
Jul 21 17:27:30.306317 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:30.305935    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8zf7t" event={"ID":"6ee3e5a6-186c-46a2-aef3-362ea08f9b30","Type":"ContainerStarted","Data":"5e370180a09c0a566d6d449786f7ddb0665a976c799e814c0d3f29f0dc53f21b"}
Jul 21 17:27:30.306317 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:30.305963    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:27:31.197700 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:31.197670    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r6qp2"
Jul 21 17:27:31.215666 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:31.215612    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8zf7t" podStartSLOduration=66.807993049 podStartE2EDuration="1m8.215594305s" totalImagesPullingTime="1.407601256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:30.327872944 +0000 UTC m=+101.899494696" watchObservedRunningTime="2026-07-21 17:27:31.215594305 +0000 UTC m=+102.787216062"
Jul 21 17:27:34.916800 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:34.916764    2562 scope.go:117] "RemoveContainer" containerID="31d61cd6c8306190dfeb09fd4a956172994c2c96b70b13f1aef83a69f1eb711f"
Jul 21 17:27:35.322369 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:35.322293    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 17:27:35.322701 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:35.322688    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/1.log"
Jul 21 17:27:35.322743 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:35.322721    2562 generic.go:356] "Generic (PLEG): container finished" podID="2302a4fe-a2b9-4de0-8dce-2805b3c2d825" containerID="ef2fef5d472bd58c14bab6660eb6badc24fef3945d66ef7ce34ec1f2b4853c45" exitCode=255
Jul 21 17:27:35.322776 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:35.322746    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-7vppx" event={"ID":"2302a4fe-a2b9-4de0-8dce-2805b3c2d825","Type":"ContainerDied","Data":"ef2fef5d472bd58c14bab6660eb6badc24fef3945d66ef7ce34ec1f2b4853c45"}
Jul 21 17:27:35.322807 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:35.322784    2562 scope.go:117] "RemoveContainer" containerID="31d61cd6c8306190dfeb09fd4a956172994c2c96b70b13f1aef83a69f1eb711f"
Jul 21 17:27:35.323114 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:35.323094    2562 scope.go:117] "RemoveContainer" containerID="ef2fef5d472bd58c14bab6660eb6badc24fef3945d66ef7ce34ec1f2b4853c45"
Jul 21 17:27:35.323328 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:35.323301    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-7vppx_openshift-console-operator(2302a4fe-a2b9-4de0-8dce-2805b3c2d825)\"" pod="openshift-console-operator/console-operator-565b7b569d-7vppx" podUID="2302a4fe-a2b9-4de0-8dce-2805b3c2d825"
Jul 21 17:27:36.326823 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:36.326794    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 17:27:38.092973 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.092943    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ccbd4944b-8z225"]
Jul 21 17:27:38.101544 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.101517    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ccbd4944b-8z225"
Jul 21 17:27:38.106325 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.106304    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4cn4b\""
Jul 21 17:27:38.106702 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.106680    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 21 17:27:38.107601 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.107581    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 21 17:27:38.107679 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.107620    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 21 17:27:38.107863 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.107851    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 21 17:27:38.117655 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.117631    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ccbd4944b-8z225"]
Jul 21 17:27:38.118768 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.118749    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc"]
Jul 21 17:27:38.121624 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.121600    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc"
Jul 21 17:27:38.124377 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.124363    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 21 17:27:38.149436 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.149411    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc"]
Jul 21 17:27:38.164646 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.164621    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bd858d955-9vzvr"]
Jul 21 17:27:38.167687 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.167653    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.171081 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.171035    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 21 17:27:38.171228 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.171139    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j5prf\""
Jul 21 17:27:38.171292 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.171250    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 21 17:27:38.171355 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.171343    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 21 17:27:38.184074 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.184040    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 21 17:27:38.198988 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.198965    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bd858d955-9vzvr"]
Jul 21 17:27:38.240777 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.240752    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhm7t\" (UniqueName: \"kubernetes.io/projected/6c1bbd5e-9c2b-4104-8a54-e82fb767a600-kube-api-access-qhm7t\") pod \"klusterlet-addon-workmgr-7bf4cb89c6-kzmpc\" (UID: \"6c1bbd5e-9c2b-4104-8a54-e82fb767a600\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc"
Jul 21 17:27:38.240937 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.240786    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqdg9\" (UniqueName: \"kubernetes.io/projected/980d52ef-e88d-45a4-bc07-4d9d780a2a4a-kube-api-access-kqdg9\") pod \"managed-serviceaccount-addon-agent-5ccbd4944b-8z225\" (UID: \"980d52ef-e88d-45a4-bc07-4d9d780a2a4a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ccbd4944b-8z225"
Jul 21 17:27:38.240937 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.240812    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/980d52ef-e88d-45a4-bc07-4d9d780a2a4a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5ccbd4944b-8z225\" (UID: \"980d52ef-e88d-45a4-bc07-4d9d780a2a4a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ccbd4944b-8z225"
Jul 21 17:27:38.240937 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.240917    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c1bbd5e-9c2b-4104-8a54-e82fb767a600-tmp\") pod \"klusterlet-addon-workmgr-7bf4cb89c6-kzmpc\" (UID: \"6c1bbd5e-9c2b-4104-8a54-e82fb767a600\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc"
Jul 21 17:27:38.241049 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.240978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6c1bbd5e-9c2b-4104-8a54-e82fb767a600-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bf4cb89c6-kzmpc\" (UID: \"6c1bbd5e-9c2b-4104-8a54-e82fb767a600\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc"
Jul 21 17:27:38.342218 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.342185    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhm7t\" (UniqueName: \"kubernetes.io/projected/6c1bbd5e-9c2b-4104-8a54-e82fb767a600-kube-api-access-qhm7t\") pod \"klusterlet-addon-workmgr-7bf4cb89c6-kzmpc\" (UID: \"6c1bbd5e-9c2b-4104-8a54-e82fb767a600\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc"
Jul 21 17:27:38.342218 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.342221    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqdg9\" (UniqueName: \"kubernetes.io/projected/980d52ef-e88d-45a4-bc07-4d9d780a2a4a-kube-api-access-kqdg9\") pod \"managed-serviceaccount-addon-agent-5ccbd4944b-8z225\" (UID: \"980d52ef-e88d-45a4-bc07-4d9d780a2a4a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ccbd4944b-8z225"
Jul 21 17:27:38.342473 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.342248    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/627270a4-71a1-4982-9798-e3d68f23eddc-ca-trust-extracted\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.342473 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.342275    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/980d52ef-e88d-45a4-bc07-4d9d780a2a4a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5ccbd4944b-8z225\" (UID: \"980d52ef-e88d-45a4-bc07-4d9d780a2a4a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ccbd4944b-8z225"
Jul 21 17:27:38.342473 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.342331    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/627270a4-71a1-4982-9798-e3d68f23eddc-installation-pull-secrets\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.342473 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.342376    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/627270a4-71a1-4982-9798-e3d68f23eddc-registry-certificates\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.342473 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.342413    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c1bbd5e-9c2b-4104-8a54-e82fb767a600-tmp\") pod \"klusterlet-addon-workmgr-7bf4cb89c6-kzmpc\" (UID: \"6c1bbd5e-9c2b-4104-8a54-e82fb767a600\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc"
Jul 21 17:27:38.342711 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.342483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/627270a4-71a1-4982-9798-e3d68f23eddc-registry-tls\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.342711 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.342526    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6c1bbd5e-9c2b-4104-8a54-e82fb767a600-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bf4cb89c6-kzmpc\" (UID: \"6c1bbd5e-9c2b-4104-8a54-e82fb767a600\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc"
Jul 21 17:27:38.342711 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.342565    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg9b5\" (UniqueName: \"kubernetes.io/projected/627270a4-71a1-4982-9798-e3d68f23eddc-kube-api-access-rg9b5\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.342711 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.342667    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/627270a4-71a1-4982-9798-e3d68f23eddc-image-registry-private-configuration\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.342711 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.342695    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/627270a4-71a1-4982-9798-e3d68f23eddc-trusted-ca\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.342886 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.342720    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/627270a4-71a1-4982-9798-e3d68f23eddc-bound-sa-token\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.342886 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.342772    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c1bbd5e-9c2b-4104-8a54-e82fb767a600-tmp\") pod \"klusterlet-addon-workmgr-7bf4cb89c6-kzmpc\" (UID: \"6c1bbd5e-9c2b-4104-8a54-e82fb767a600\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc"
Jul 21 17:27:38.344983 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.344926    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6c1bbd5e-9c2b-4104-8a54-e82fb767a600-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bf4cb89c6-kzmpc\" (UID: \"6c1bbd5e-9c2b-4104-8a54-e82fb767a600\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc"
Jul 21 17:27:38.345103 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.345031    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/980d52ef-e88d-45a4-bc07-4d9d780a2a4a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5ccbd4944b-8z225\" (UID: \"980d52ef-e88d-45a4-bc07-4d9d780a2a4a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ccbd4944b-8z225"
Jul 21 17:27:38.365607 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.365578    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhm7t\" (UniqueName: \"kubernetes.io/projected/6c1bbd5e-9c2b-4104-8a54-e82fb767a600-kube-api-access-qhm7t\") pod \"klusterlet-addon-workmgr-7bf4cb89c6-kzmpc\" (UID: \"6c1bbd5e-9c2b-4104-8a54-e82fb767a600\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc"
Jul 21 17:27:38.373862 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.373840    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqdg9\" (UniqueName: \"kubernetes.io/projected/980d52ef-e88d-45a4-bc07-4d9d780a2a4a-kube-api-access-kqdg9\") pod \"managed-serviceaccount-addon-agent-5ccbd4944b-8z225\" (UID: \"980d52ef-e88d-45a4-bc07-4d9d780a2a4a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ccbd4944b-8z225"
Jul 21 17:27:38.420362 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.420327    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ccbd4944b-8z225"
Jul 21 17:27:38.429492 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.429469    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc"
Jul 21 17:27:38.443695 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.443668    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/627270a4-71a1-4982-9798-e3d68f23eddc-installation-pull-secrets\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.443695 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.443702    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/627270a4-71a1-4982-9798-e3d68f23eddc-registry-certificates\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.443877 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.443767    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/627270a4-71a1-4982-9798-e3d68f23eddc-registry-tls\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.443877 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.443805    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg9b5\" (UniqueName: \"kubernetes.io/projected/627270a4-71a1-4982-9798-e3d68f23eddc-kube-api-access-rg9b5\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.443877 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.443843    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/627270a4-71a1-4982-9798-e3d68f23eddc-image-registry-private-configuration\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.443877 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.443869    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/627270a4-71a1-4982-9798-e3d68f23eddc-trusted-ca\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.444051 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.443896    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/627270a4-71a1-4982-9798-e3d68f23eddc-bound-sa-token\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.444051 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.443928    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/627270a4-71a1-4982-9798-e3d68f23eddc-ca-trust-extracted\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.444753 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.444638    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/627270a4-71a1-4982-9798-e3d68f23eddc-ca-trust-extracted\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.445053 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.445019    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/627270a4-71a1-4982-9798-e3d68f23eddc-registry-certificates\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.445290 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.445238    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/627270a4-71a1-4982-9798-e3d68f23eddc-trusted-ca\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.446773 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.446751    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/627270a4-71a1-4982-9798-e3d68f23eddc-registry-tls\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.446887 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.446806    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/627270a4-71a1-4982-9798-e3d68f23eddc-installation-pull-secrets\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.447562 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.447540    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/627270a4-71a1-4982-9798-e3d68f23eddc-image-registry-private-configuration\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.457453 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.457420    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg9b5\" (UniqueName: \"kubernetes.io/projected/627270a4-71a1-4982-9798-e3d68f23eddc-kube-api-access-rg9b5\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.470002 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.469975    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/627270a4-71a1-4982-9798-e3d68f23eddc-bound-sa-token\") pod \"image-registry-6bd858d955-9vzvr\" (UID: \"627270a4-71a1-4982-9798-e3d68f23eddc\") " pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.478295 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.477763    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:38.581816 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.581788    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ccbd4944b-8z225"]
Jul 21 17:27:38.583342 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:27:38.583315    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod980d52ef_e88d_45a4_bc07_4d9d780a2a4a.slice/crio-f0d86188f2173735d5bc746d28ad2e7a3d6039030e279b5f731bc7950e812a24 WatchSource:0}: Error finding container f0d86188f2173735d5bc746d28ad2e7a3d6039030e279b5f731bc7950e812a24: Status 404 returned error can't find the container with id f0d86188f2173735d5bc746d28ad2e7a3d6039030e279b5f731bc7950e812a24
Jul 21 17:27:38.604160 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.604057    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc"]
Jul 21 17:27:38.606420 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:27:38.606390    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1bbd5e_9c2b_4104_8a54_e82fb767a600.slice/crio-b02b4c7dd4e309821d47139e4233d660d250aad7f61afb5416cdbba39baf3658 WatchSource:0}: Error finding container b02b4c7dd4e309821d47139e4233d660d250aad7f61afb5416cdbba39baf3658: Status 404 returned error can't find the container with id b02b4c7dd4e309821d47139e4233d660d250aad7f61afb5416cdbba39baf3658
Jul 21 17:27:38.650184 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:38.650143    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bd858d955-9vzvr"]
Jul 21 17:27:38.653909 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:27:38.653890    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627270a4_71a1_4982_9798_e3d68f23eddc.slice/crio-568c10708b4bc9f5b628ba585332884fe100c004fb5e57f28f60b21f810394ea WatchSource:0}: Error finding container 568c10708b4bc9f5b628ba585332884fe100c004fb5e57f28f60b21f810394ea: Status 404 returned error can't find the container with id 568c10708b4bc9f5b628ba585332884fe100c004fb5e57f28f60b21f810394ea
Jul 21 17:27:39.339102 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:39.338783    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bd858d955-9vzvr" event={"ID":"627270a4-71a1-4982-9798-e3d68f23eddc","Type":"ContainerStarted","Data":"10e453e8a5c04b7a00a69abfa9e42a3c53ac27100b8e60ea4be66c82be825765"}
Jul 21 17:27:39.339102 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:39.338871    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bd858d955-9vzvr" event={"ID":"627270a4-71a1-4982-9798-e3d68f23eddc","Type":"ContainerStarted","Data":"568c10708b4bc9f5b628ba585332884fe100c004fb5e57f28f60b21f810394ea"}
Jul 21 17:27:39.339102 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:39.338899    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:27:39.340353 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:39.340321    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ccbd4944b-8z225" event={"ID":"980d52ef-e88d-45a4-bc07-4d9d780a2a4a","Type":"ContainerStarted","Data":"f0d86188f2173735d5bc746d28ad2e7a3d6039030e279b5f731bc7950e812a24"}
Jul 21 17:27:39.341606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:39.341587    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc" event={"ID":"6c1bbd5e-9c2b-4104-8a54-e82fb767a600","Type":"ContainerStarted","Data":"b02b4c7dd4e309821d47139e4233d660d250aad7f61afb5416cdbba39baf3658"}
Jul 21 17:27:39.367341 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:39.367281    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6bd858d955-9vzvr" podStartSLOduration=1.367261099 podStartE2EDuration="1.367261099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:39.365760007 +0000 UTC m=+110.937381790" watchObservedRunningTime="2026-07-21 17:27:39.367261099 +0000 UTC m=+110.938882852"
Jul 21 17:27:39.453008 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:39.452968    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:39.456176 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:39.456128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5757dbc1-e93d-4fcd-b888-c3e7cb4955b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6btrk\" (UID: \"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2\") " pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:39.741434 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:39.741399    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6btrk"
Jul 21 17:27:39.910973 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:39.910942    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6btrk"]
Jul 21 17:27:40.312549 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:40.312511    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8zf7t"
Jul 21 17:27:40.681131 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:27:40.681089    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5757dbc1_e93d_4fcd_b888_c3e7cb4955b2.slice/crio-e24298bbdd112eba47b9f618cdb9a4dc22a9080e64d2bd638bd48cac52d81a89 WatchSource:0}: Error finding container e24298bbdd112eba47b9f618cdb9a4dc22a9080e64d2bd638bd48cac52d81a89: Status 404 returned error can't find the container with id e24298bbdd112eba47b9f618cdb9a4dc22a9080e64d2bd638bd48cac52d81a89
Jul 21 17:27:40.823659 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:40.823624    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:40.823820 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:40.823705    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:27:40.824143 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:40.824123    2562 scope.go:117] "RemoveContainer" containerID="ef2fef5d472bd58c14bab6660eb6badc24fef3945d66ef7ce34ec1f2b4853c45"
Jul 21 17:27:40.824352 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:40.824322    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-7vppx_openshift-console-operator(2302a4fe-a2b9-4de0-8dce-2805b3c2d825)\"" pod="openshift-console-operator/console-operator-565b7b569d-7vppx" podUID="2302a4fe-a2b9-4de0-8dce-2805b3c2d825"
Jul 21 17:27:41.351625 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:41.351580    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6btrk" event={"ID":"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2","Type":"ContainerStarted","Data":"e24298bbdd112eba47b9f618cdb9a4dc22a9080e64d2bd638bd48cac52d81a89"}
Jul 21 17:27:42.359799 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:42.359719    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc" event={"ID":"6c1bbd5e-9c2b-4104-8a54-e82fb767a600","Type":"ContainerStarted","Data":"3795b8b62f868fa8b1d7e1e826627a122d3aea172a56e45de3cb95279665f171"}
Jul 21 17:27:42.360387 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:42.360367    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc"
Jul 21 17:27:42.362117 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:42.362093    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc"
Jul 21 17:27:42.381734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:42.381248    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bf4cb89c6-kzmpc" podStartSLOduration=0.925480423 podStartE2EDuration="4.381229921s" totalImagesPullingTime="3.455749498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:42.380421951 +0000 UTC m=+113.952043707" watchObservedRunningTime="2026-07-21 17:27:42.381229921 +0000 UTC m=+113.952851677"
Jul 21 17:27:43.367488 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:43.367445    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ccbd4944b-8z225" event={"ID":"980d52ef-e88d-45a4-bc07-4d9d780a2a4a","Type":"ContainerStarted","Data":"08b488f5cc0a9a018eea61e20dfd785520c70f16c75d12e947a417cb21201a70"}
Jul 21 17:27:43.384945 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:43.384892    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ccbd4944b-8z225" podStartSLOduration=0.711055768 podStartE2EDuration="5.384873894s" totalImagesPullingTime="4.673818126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:43.383778496 +0000 UTC m=+114.955400252" watchObservedRunningTime="2026-07-21 17:27:43.384873894 +0000 UTC m=+114.956495689"
Jul 21 17:27:47.381662 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.381556    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zdbqk"]
Jul 21 17:27:47.384834 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.384810    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6btrk" event={"ID":"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2","Type":"ContainerStarted","Data":"244759f0b79f1e423f4c358bf64e950d615ff635268b8c9dc30403409eb23369"}
Jul 21 17:27:47.384954 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.384904    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.387675 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.387655    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 21 17:27:47.387782 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.387686    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 21 17:27:47.387782 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.387694    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 21 17:27:47.388206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.388191    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t76wx\""
Jul 21 17:27:47.388298 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.388283    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 21 17:27:47.388374 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.388361    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 21 17:27:47.388781 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.388768    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 21 17:27:47.421991 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.421966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27b86f8c-114a-4875-ac3f-29fa472ca8ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.422177 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.421995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27b86f8c-114a-4875-ac3f-29fa472ca8ef-node-exporter-tls\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.422177 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.422014    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-896rg\" (UniqueName: \"kubernetes.io/projected/27b86f8c-114a-4875-ac3f-29fa472ca8ef-kube-api-access-896rg\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.422177 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.422075    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27b86f8c-114a-4875-ac3f-29fa472ca8ef-node-exporter-wtmp\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.422177 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.422102    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27b86f8c-114a-4875-ac3f-29fa472ca8ef-metrics-client-ca\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.422383 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.422198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27b86f8c-114a-4875-ac3f-29fa472ca8ef-root\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.422383 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.422236    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27b86f8c-114a-4875-ac3f-29fa472ca8ef-sys\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.422383 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.422263    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27b86f8c-114a-4875-ac3f-29fa472ca8ef-node-exporter-textfile\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.422383 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.422290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27b86f8c-114a-4875-ac3f-29fa472ca8ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.522868 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.522837    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27b86f8c-114a-4875-ac3f-29fa472ca8ef-root\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.522868 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.522871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27b86f8c-114a-4875-ac3f-29fa472ca8ef-sys\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.523143 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.522888    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27b86f8c-114a-4875-ac3f-29fa472ca8ef-node-exporter-textfile\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.523143 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.522906    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27b86f8c-114a-4875-ac3f-29fa472ca8ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.523143 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.522949    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27b86f8c-114a-4875-ac3f-29fa472ca8ef-root\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.523143 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.522961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27b86f8c-114a-4875-ac3f-29fa472ca8ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.523143 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.522968    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27b86f8c-114a-4875-ac3f-29fa472ca8ef-sys\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.523143 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.522986    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27b86f8c-114a-4875-ac3f-29fa472ca8ef-node-exporter-tls\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.523143 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.523015    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-896rg\" (UniqueName: \"kubernetes.io/projected/27b86f8c-114a-4875-ac3f-29fa472ca8ef-kube-api-access-896rg\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.523143 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.523049    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27b86f8c-114a-4875-ac3f-29fa472ca8ef-node-exporter-wtmp\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.523143 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.523097    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27b86f8c-114a-4875-ac3f-29fa472ca8ef-metrics-client-ca\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.523554 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.523311    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27b86f8c-114a-4875-ac3f-29fa472ca8ef-node-exporter-textfile\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.523554 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.523395    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27b86f8c-114a-4875-ac3f-29fa472ca8ef-node-exporter-wtmp\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.523691 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.523621    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27b86f8c-114a-4875-ac3f-29fa472ca8ef-metrics-client-ca\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.523691 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.523664    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27b86f8c-114a-4875-ac3f-29fa472ca8ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.525545 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.525510    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27b86f8c-114a-4875-ac3f-29fa472ca8ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.525773 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.525755    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27b86f8c-114a-4875-ac3f-29fa472ca8ef-node-exporter-tls\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.531758 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.531736    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-896rg\" (UniqueName: \"kubernetes.io/projected/27b86f8c-114a-4875-ac3f-29fa472ca8ef-kube-api-access-896rg\") pod \"node-exporter-zdbqk\" (UID: \"27b86f8c-114a-4875-ac3f-29fa472ca8ef\") " pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:47.693736 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:47.693692    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zdbqk"
Jul 21 17:27:48.390084 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:48.390017    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdbqk" event={"ID":"27b86f8c-114a-4875-ac3f-29fa472ca8ef","Type":"ContainerStarted","Data":"4746160c37f7773d83e08db60173f132b6339ff94189200f04ee4d2a90b17e05"}
Jul 21 17:27:49.285183 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.285143    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-68bc74c496-s4xs8"]
Jul 21 17:27:49.295475 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.295449    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.298359 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.298335    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 21 17:27:49.298748 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.298370    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 21 17:27:49.298748 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.298518    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 21 17:27:49.298748 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.298383    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e50uoe9h6j2f7\""
Jul 21 17:27:49.298748 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.298388    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 21 17:27:49.298748 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.298410    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-x58zb\""
Jul 21 17:27:49.298748 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.298642    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 21 17:27:49.301709 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.301692    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68bc74c496-s4xs8"]
Jul 21 17:27:49.336661 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.336625    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.336846 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.336677    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94326f12-be7e-45fc-a6c3-8e1709c96b02-metrics-client-ca\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.336846 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.336756    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.336846 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.336798    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p299k\" (UniqueName: \"kubernetes.io/projected/94326f12-be7e-45fc-a6c3-8e1709c96b02-kube-api-access-p299k\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.336976 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.336892    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-thanos-querier-tls\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.336976 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.336918    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.336976 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.336944    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.337139 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.337037    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-grpc-tls\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.395162 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.395123    2562 generic.go:356] "Generic (PLEG): container finished" podID="27b86f8c-114a-4875-ac3f-29fa472ca8ef" containerID="da0bb3d08d10131759fabfc5db050c5250cffadee96568c4f5fc3b4561cf427a" exitCode=0
Jul 21 17:27:49.395588 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.395193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdbqk" event={"ID":"27b86f8c-114a-4875-ac3f-29fa472ca8ef","Type":"ContainerDied","Data":"da0bb3d08d10131759fabfc5db050c5250cffadee96568c4f5fc3b4561cf427a"}
Jul 21 17:27:49.437965 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.437935    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94326f12-be7e-45fc-a6c3-8e1709c96b02-metrics-client-ca\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.438149 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.437998    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.438149 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.438046    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p299k\" (UniqueName: \"kubernetes.io/projected/94326f12-be7e-45fc-a6c3-8e1709c96b02-kube-api-access-p299k\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.438274 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.438147    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-thanos-querier-tls\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.438274 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.438174    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.438274 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.438201    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.438475 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.438293    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-grpc-tls\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.438475 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.438348    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.438746 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.438719    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94326f12-be7e-45fc-a6c3-8e1709c96b02-metrics-client-ca\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.441347 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.441319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.441435 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.441360    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-grpc-tls\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.441509 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.441430    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-thanos-querier-tls\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.441562 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.441515    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.441782 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.441762    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.441860 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.441788    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94326f12-be7e-45fc-a6c3-8e1709c96b02-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.448918 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.448898    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p299k\" (UniqueName: \"kubernetes.io/projected/94326f12-be7e-45fc-a6c3-8e1709c96b02-kube-api-access-p299k\") pod \"thanos-querier-68bc74c496-s4xs8\" (UID: \"94326f12-be7e-45fc-a6c3-8e1709c96b02\") " pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:49.606961 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:49.606876    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:51.100451 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:51.100372    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68bc74c496-s4xs8"]
Jul 21 17:27:51.103263 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:27:51.103234    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94326f12_be7e_45fc_a6c3_8e1709c96b02.slice/crio-20e972895d267f12bc57bfa5a9028d97eaedd4a1d590df654bde97330a4aff38 WatchSource:0}: Error finding container 20e972895d267f12bc57bfa5a9028d97eaedd4a1d590df654bde97330a4aff38: Status 404 returned error can't find the container with id 20e972895d267f12bc57bfa5a9028d97eaedd4a1d590df654bde97330a4aff38
Jul 21 17:27:51.403979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:51.403944    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6btrk" event={"ID":"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2","Type":"ContainerStarted","Data":"953a65b12dae68f81145b523b09b57acd04181b447823c8e4cc2506ef109b031"}
Jul 21 17:27:51.405128 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:51.405104    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8" event={"ID":"94326f12-be7e-45fc-a6c3-8e1709c96b02","Type":"ContainerStarted","Data":"20e972895d267f12bc57bfa5a9028d97eaedd4a1d590df654bde97330a4aff38"}
Jul 21 17:27:51.406971 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:51.406953    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdbqk" event={"ID":"27b86f8c-114a-4875-ac3f-29fa472ca8ef","Type":"ContainerStarted","Data":"edb145bcfc340f95fe3eb4573ff24a92c0a1ba4df414650aa9a42091a1b56b08"}
Jul 21 17:27:51.407050 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:51.406996    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdbqk" event={"ID":"27b86f8c-114a-4875-ac3f-29fa472ca8ef","Type":"ContainerStarted","Data":"29f3bb43bd63b456eacb9ea640bfe333eee369ebdc2a52b7473407821d9100a3"}
Jul 21 17:27:51.941896 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:51.941723    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zdbqk" podStartSLOduration=4.004225846 podStartE2EDuration="4.941699781s" totalImagesPullingTime="857.522923ms" totalInitContainerRuntime="79.951012ms" isStatefulPod=true podCreationTimestamp="2026-07-21 17:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:51.43128232 +0000 UTC m=+123.002904096" watchObservedRunningTime="2026-07-21 17:27:51.941699781 +0000 UTC m=+123.513321536"
Jul 21 17:27:51.943006 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:51.942980    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv"]
Jul 21 17:27:51.946488 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:51.946464    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv"
Jul 21 17:27:51.949336 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:51.949309    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 21 17:27:51.949440 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:51.949338    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-q6p7p\""
Jul 21 17:27:51.958986 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:51.958956    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0b85072-48eb-4ac9-a1a4-90baa59d26c4-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-sjjsv\" (UID: \"b0b85072-48eb-4ac9-a1a4-90baa59d26c4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv"
Jul 21 17:27:51.959107 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:51.959018    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnscq\" (UniqueName: \"kubernetes.io/projected/b0b85072-48eb-4ac9-a1a4-90baa59d26c4-kube-api-access-rnscq\") pod \"monitoring-plugin-7b7b7bc8bf-sjjsv\" (UID: \"b0b85072-48eb-4ac9-a1a4-90baa59d26c4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv"
Jul 21 17:27:51.959107 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:51.959056    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv"]
Jul 21 17:27:52.059987 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:52.059945    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnscq\" (UniqueName: \"kubernetes.io/projected/b0b85072-48eb-4ac9-a1a4-90baa59d26c4-kube-api-access-rnscq\") pod \"monitoring-plugin-7b7b7bc8bf-sjjsv\" (UID: \"b0b85072-48eb-4ac9-a1a4-90baa59d26c4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv"
Jul 21 17:27:52.060181 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:52.060085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0b85072-48eb-4ac9-a1a4-90baa59d26c4-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-sjjsv\" (UID: \"b0b85072-48eb-4ac9-a1a4-90baa59d26c4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv"
Jul 21 17:27:52.060254 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:52.060208    2562 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 21 17:27:52.060305 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:52.060268    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b85072-48eb-4ac9-a1a4-90baa59d26c4-monitoring-plugin-cert podName:b0b85072-48eb-4ac9-a1a4-90baa59d26c4 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:52.560248686 +0000 UTC m=+124.131870426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/b0b85072-48eb-4ac9-a1a4-90baa59d26c4-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-sjjsv" (UID: "b0b85072-48eb-4ac9-a1a4-90baa59d26c4") : secret "monitoring-plugin-cert" not found
Jul 21 17:27:52.072189 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:52.072135    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnscq\" (UniqueName: \"kubernetes.io/projected/b0b85072-48eb-4ac9-a1a4-90baa59d26c4-kube-api-access-rnscq\") pod \"monitoring-plugin-7b7b7bc8bf-sjjsv\" (UID: \"b0b85072-48eb-4ac9-a1a4-90baa59d26c4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv"
Jul 21 17:27:52.565455 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:52.565422    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0b85072-48eb-4ac9-a1a4-90baa59d26c4-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-sjjsv\" (UID: \"b0b85072-48eb-4ac9-a1a4-90baa59d26c4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv"
Jul 21 17:27:52.568671 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:52.568642    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b0b85072-48eb-4ac9-a1a4-90baa59d26c4-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-sjjsv\" (UID: \"b0b85072-48eb-4ac9-a1a4-90baa59d26c4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv"
Jul 21 17:27:52.858842 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:52.858818    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv"
Jul 21 17:27:53.010383 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:53.010196    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv"]
Jul 21 17:27:53.416650 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:53.416616    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8" event={"ID":"94326f12-be7e-45fc-a6c3-8e1709c96b02","Type":"ContainerStarted","Data":"f0c8351e14fbbf3c1b4e51526c43677016e849f36f7ff3ba97dfc04250240967"}
Jul 21 17:27:54.654563 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:27:54.654531    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b85072_48eb_4ac9_a1a4_90baa59d26c4.slice/crio-54da482806f85eeeb78b7d503d6ba9017d0dc7ff1558415e17147568c059ec76 WatchSource:0}: Error finding container 54da482806f85eeeb78b7d503d6ba9017d0dc7ff1558415e17147568c059ec76: Status 404 returned error can't find the container with id 54da482806f85eeeb78b7d503d6ba9017d0dc7ff1558415e17147568c059ec76
Jul 21 17:27:54.916763 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:54.916685    2562 scope.go:117] "RemoveContainer" containerID="ef2fef5d472bd58c14bab6660eb6badc24fef3945d66ef7ce34ec1f2b4853c45"
Jul 21 17:27:54.916909 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:27:54.916849    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-7vppx_openshift-console-operator(2302a4fe-a2b9-4de0-8dce-2805b3c2d825)\"" pod="openshift-console-operator/console-operator-565b7b569d-7vppx" podUID="2302a4fe-a2b9-4de0-8dce-2805b3c2d825"
Jul 21 17:27:55.426830 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:55.426786    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv" event={"ID":"b0b85072-48eb-4ac9-a1a4-90baa59d26c4","Type":"ContainerStarted","Data":"54da482806f85eeeb78b7d503d6ba9017d0dc7ff1558415e17147568c059ec76"}
Jul 21 17:27:55.430004 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:55.429966    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6btrk" event={"ID":"5757dbc1-e93d-4fcd-b888-c3e7cb4955b2","Type":"ContainerStarted","Data":"be5730765dfbf165c92b17fb2e78d0f86a197e664cca2f9d1c514de9e247e7ed"}
Jul 21 17:27:55.432367 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:55.432338    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8" event={"ID":"94326f12-be7e-45fc-a6c3-8e1709c96b02","Type":"ContainerStarted","Data":"31a121f105ac761694c659aa60445bc51d2a0deabec757c999a66e01c1aae49c"}
Jul 21 17:27:55.432525 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:55.432414    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8" event={"ID":"94326f12-be7e-45fc-a6c3-8e1709c96b02","Type":"ContainerStarted","Data":"2277d13155cf29301158a80952a4b415f0ba2a3cad7781bfacd2217818738d5a"}
Jul 21 17:27:55.454930 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:55.454866    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6btrk" podStartSLOduration=34.563340233 podStartE2EDuration="48.454841939s" totalImagesPullingTime="13.891501706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:27:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:55.45346174 +0000 UTC m=+127.025083500" watchObservedRunningTime="2026-07-21 17:27:55.454841939 +0000 UTC m=+127.026463694"
Jul 21 17:27:56.438962 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:56.438922    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8" event={"ID":"94326f12-be7e-45fc-a6c3-8e1709c96b02","Type":"ContainerStarted","Data":"84aead8c114391d574013e0219aa1e306fcbc69784d21465d1ca70e0dbfd8575"}
Jul 21 17:27:56.439456 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:56.439001    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8" event={"ID":"94326f12-be7e-45fc-a6c3-8e1709c96b02","Type":"ContainerStarted","Data":"4cedcba93db73676ec498b12773173a0ec07fb028c83a9697369ad3f555774bf"}
Jul 21 17:27:56.439456 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:56.439047    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:27:56.439456 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:56.439107    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8" event={"ID":"94326f12-be7e-45fc-a6c3-8e1709c96b02","Type":"ContainerStarted","Data":"50ce6dcb231571d2a4e70d2866f7038eae1cb39a089d74a73607778b5dde7a60"}
Jul 21 17:27:56.440477 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:56.440452    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv" event={"ID":"b0b85072-48eb-4ac9-a1a4-90baa59d26c4","Type":"ContainerStarted","Data":"067e1b231c5b3d865c6e209db4f951cb9d2c4ddf6a3403d284fefe79ea17e4b4"}
Jul 21 17:27:56.440611 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:56.440493    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv"
Jul 21 17:27:56.445846 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:56.445823    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv"
Jul 21 17:27:56.465388 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:56.465324    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8" podStartSLOduration=4.698530096 podStartE2EDuration="7.465300909s" totalImagesPullingTime="2.766770813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:56.464377904 +0000 UTC m=+128.035999680" watchObservedRunningTime="2026-07-21 17:27:56.465300909 +0000 UTC m=+128.036922661"
Jul 21 17:27:58.719654 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:58.719619    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs\") pod \"network-metrics-daemon-9r9z4\" (UID: \"cb83c0b0-6020-43bf-be65-b41d51b76e5a\") " pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:27:58.722205 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:58.722172    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb83c0b0-6020-43bf-be65-b41d51b76e5a-metrics-certs\") pod \"network-metrics-daemon-9r9z4\" (UID: \"cb83c0b0-6020-43bf-be65-b41d51b76e5a\") " pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:27:58.730738 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:58.730713    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-swxfv\""
Jul 21 17:27:58.739059 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:58.739041    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9r9z4"
Jul 21 17:27:58.866359 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:58.866306    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-sjjsv" podStartSLOduration=6.669052295 podStartE2EDuration="7.866288651s" totalImagesPullingTime="1.197236356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:56.4806864 +0000 UTC m=+128.052308190" watchObservedRunningTime="2026-07-21 17:27:58.866288651 +0000 UTC m=+130.437910406"
Jul 21 17:27:58.866805 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:58.866785    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9r9z4"]
Jul 21 17:27:58.869728 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:27:58.869705    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb83c0b0_6020_43bf_be65_b41d51b76e5a.slice/crio-1c5cd290844aea071a8baaef0195f02b389dc7e0b49941d36dd9e566540ce5ff WatchSource:0}: Error finding container 1c5cd290844aea071a8baaef0195f02b389dc7e0b49941d36dd9e566540ce5ff: Status 404 returned error can't find the container with id 1c5cd290844aea071a8baaef0195f02b389dc7e0b49941d36dd9e566540ce5ff
Jul 21 17:27:59.452958 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:27:59.452914    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9r9z4" event={"ID":"cb83c0b0-6020-43bf-be65-b41d51b76e5a","Type":"ContainerStarted","Data":"1c5cd290844aea071a8baaef0195f02b389dc7e0b49941d36dd9e566540ce5ff"}
Jul 21 17:28:00.350542 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:00.350466    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bd858d955-9vzvr"
Jul 21 17:28:00.457891 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:00.457851    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9r9z4" event={"ID":"cb83c0b0-6020-43bf-be65-b41d51b76e5a","Type":"ContainerStarted","Data":"4777bde9e586c3b8597d26d65515ec5df99ce6023739db6943db64d5b7ac291f"}
Jul 21 17:28:00.458087 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:00.457904    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9r9z4" event={"ID":"cb83c0b0-6020-43bf-be65-b41d51b76e5a","Type":"ContainerStarted","Data":"fbd17c77c0b951925f570e459de73284542dfc925adcdd77de4f2ccc20256959"}
Jul 21 17:28:00.476445 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:00.476393    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9r9z4" podStartSLOduration=131.407019126 podStartE2EDuration="2m12.47637801s" totalImagesPullingTime="1.069358884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:28:00.475154444 +0000 UTC m=+132.046776207" watchObservedRunningTime="2026-07-21 17:28:00.47637801 +0000 UTC m=+132.047999764"
Jul 21 17:28:02.451458 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:02.451427    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-68bc74c496-s4xs8"
Jul 21 17:28:06.916614 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:06.916566    2562 scope.go:117] "RemoveContainer" containerID="ef2fef5d472bd58c14bab6660eb6badc24fef3945d66ef7ce34ec1f2b4853c45"
Jul 21 17:28:07.485991 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:07.485959    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 17:28:07.486180 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:07.486011    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-7vppx" event={"ID":"2302a4fe-a2b9-4de0-8dce-2805b3c2d825","Type":"ContainerStarted","Data":"ee4ed69372478056f7ab6f7015c71549d7a83d0d57243f00fb6e9b8f33508b42"}
Jul 21 17:28:07.486307 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:07.486291    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:28:07.511994 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:07.511937    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-7vppx" podStartSLOduration=55.391819466 podStartE2EDuration="57.511922171s" totalImagesPullingTime="2.120102705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:28:07.511358447 +0000 UTC m=+139.082980222" watchObservedRunningTime="2026-07-21 17:28:07.511922171 +0000 UTC m=+139.083543925"
Jul 21 17:28:07.629374 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:07.629210    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-7vppx"
Jul 21 17:28:19.281755 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.281722    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54dc988fb9-bqp2l"]
Jul 21 17:28:19.442578 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.442539    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54dc988fb9-bqp2l"]
Jul 21 17:28:19.442741 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.442643    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.446481 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.446454    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 21 17:28:19.446642 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.446628    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 21 17:28:19.446887 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.446873    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 21 17:28:19.446948 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.446913    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 21 17:28:19.448358 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.448335    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 21 17:28:19.448358 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.448348    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 21 17:28:19.448534 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.448341    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 21 17:28:19.448702 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.448686    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-426zn\""
Jul 21 17:28:19.582496 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.582422    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-oauth-serving-cert\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.582496 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.582462    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-console-config\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.582496 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.582478    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-service-ca\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.582700 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.582551    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f22d5b11-93c8-4397-992c-441324d1a24f-console-oauth-config\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.582700 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.582607    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8vkc\" (UniqueName: \"kubernetes.io/projected/f22d5b11-93c8-4397-992c-441324d1a24f-kube-api-access-s8vkc\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.582700 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.582632    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f22d5b11-93c8-4397-992c-441324d1a24f-console-serving-cert\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.683767 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.683728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f22d5b11-93c8-4397-992c-441324d1a24f-console-serving-cert\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.683973 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.683793    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-oauth-serving-cert\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.683973 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.683827    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-console-config\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.683973 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.683842    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-service-ca\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.683973 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.683875    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f22d5b11-93c8-4397-992c-441324d1a24f-console-oauth-config\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.683973 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.683903    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vkc\" (UniqueName: \"kubernetes.io/projected/f22d5b11-93c8-4397-992c-441324d1a24f-kube-api-access-s8vkc\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.684658 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.684630    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-console-config\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.684766 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.684703    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-service-ca\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.684766 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.684750    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-oauth-serving-cert\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.686534 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.686511    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f22d5b11-93c8-4397-992c-441324d1a24f-console-oauth-config\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.686673 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.686655    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f22d5b11-93c8-4397-992c-441324d1a24f-console-serving-cert\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.694225 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.694201    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8vkc\" (UniqueName: \"kubernetes.io/projected/f22d5b11-93c8-4397-992c-441324d1a24f-kube-api-access-s8vkc\") pod \"console-54dc988fb9-bqp2l\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") " pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.751793 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.751762    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:19.926822 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:19.926798    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54dc988fb9-bqp2l"]
Jul 21 17:28:19.927561 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:28:19.927532    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22d5b11_93c8_4397_992c_441324d1a24f.slice/crio-02d8c4e2f5c66569063c54686d0324bde4d9f2aeba4b19fb80c784c2774f42ff WatchSource:0}: Error finding container 02d8c4e2f5c66569063c54686d0324bde4d9f2aeba4b19fb80c784c2774f42ff: Status 404 returned error can't find the container with id 02d8c4e2f5c66569063c54686d0324bde4d9f2aeba4b19fb80c784c2774f42ff
Jul 21 17:28:20.539278 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:20.539203    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54dc988fb9-bqp2l" event={"ID":"f22d5b11-93c8-4397-992c-441324d1a24f","Type":"ContainerStarted","Data":"02d8c4e2f5c66569063c54686d0324bde4d9f2aeba4b19fb80c784c2774f42ff"}
Jul 21 17:28:23.552013 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:23.551976    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54dc988fb9-bqp2l" event={"ID":"f22d5b11-93c8-4397-992c-441324d1a24f","Type":"ContainerStarted","Data":"bfd24e2146556107311d761c140f0854b0f9060421deac050038a378d150ad21"}
Jul 21 17:28:23.553282 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:23.553258    2562 generic.go:356] "Generic (PLEG): container finished" podID="0a37a390-191f-4644-b398-6dcf97268ee8" containerID="85e1f309d706aa2ca83e628669ba5503a0b83ce189eeed309798d5772db077da" exitCode=0
Jul 21 17:28:23.553415 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:23.553305    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-9qvnl" event={"ID":"0a37a390-191f-4644-b398-6dcf97268ee8","Type":"ContainerDied","Data":"85e1f309d706aa2ca83e628669ba5503a0b83ce189eeed309798d5772db077da"}
Jul 21 17:28:23.553609 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:23.553594    2562 scope.go:117] "RemoveContainer" containerID="85e1f309d706aa2ca83e628669ba5503a0b83ce189eeed309798d5772db077da"
Jul 21 17:28:23.572789 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:23.572741    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54dc988fb9-bqp2l" podStartSLOduration=1.719328288 podStartE2EDuration="4.572725229s" totalImagesPullingTime="2.853396941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:28:23.571711985 +0000 UTC m=+155.143333750" watchObservedRunningTime="2026-07-21 17:28:23.572725229 +0000 UTC m=+155.144346983"
Jul 21 17:28:24.559363 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:24.559322    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-9qvnl" event={"ID":"0a37a390-191f-4644-b398-6dcf97268ee8","Type":"ContainerStarted","Data":"f2e4d6d341c6a4c87ff2207277d9ea3a6a383fe202e277b4b0812b767ac71f57"}
Jul 21 17:28:27.371437 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:27.371408    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-sjjsv_b0b85072-48eb-4ac9-a1a4-90baa59d26c4/monitoring-plugin/0.log"
Jul 21 17:28:28.770352 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:28.770322    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zdbqk_27b86f8c-114a-4875-ac3f-29fa472ca8ef/init-textfile/0.log"
Jul 21 17:28:28.971877 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:28.971847    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zdbqk_27b86f8c-114a-4875-ac3f-29fa472ca8ef/node-exporter/0.log"
Jul 21 17:28:29.171007 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:29.170976    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zdbqk_27b86f8c-114a-4875-ac3f-29fa472ca8ef/kube-rbac-proxy/0.log"
Jul 21 17:28:29.751889 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:29.751863    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:29.752059 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:29.751925    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:29.756878 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:29.756851    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:30.586851 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:30.586824    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:28:32.571352 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:32.571323    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bc74c496-s4xs8_94326f12-be7e-45fc-a6c3-8e1709c96b02/thanos-query/0.log"
Jul 21 17:28:32.771123 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:32.771060    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bc74c496-s4xs8_94326f12-be7e-45fc-a6c3-8e1709c96b02/kube-rbac-proxy-web/0.log"
Jul 21 17:28:32.970481 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:32.970447    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bc74c496-s4xs8_94326f12-be7e-45fc-a6c3-8e1709c96b02/kube-rbac-proxy/0.log"
Jul 21 17:28:33.170813 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:33.170785    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bc74c496-s4xs8_94326f12-be7e-45fc-a6c3-8e1709c96b02/prom-label-proxy/0.log"
Jul 21 17:28:33.371726 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:33.371652    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bc74c496-s4xs8_94326f12-be7e-45fc-a6c3-8e1709c96b02/kube-rbac-proxy-rules/0.log"
Jul 21 17:28:33.570920 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:33.570887    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bc74c496-s4xs8_94326f12-be7e-45fc-a6c3-8e1709c96b02/kube-rbac-proxy-metrics/0.log"
Jul 21 17:28:33.970969 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:33.970940    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 17:28:34.172004 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:34.171975    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/3.log"
Jul 21 17:28:34.371203 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:34.371132    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-54dc988fb9-bqp2l_f22d5b11-93c8-4397-992c-441324d1a24f/console/0.log"
Jul 21 17:28:34.771398 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:34.771368    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8zf7t_6ee3e5a6-186c-46a2-aef3-362ea08f9b30/dns/0.log"
Jul 21 17:28:34.970389 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:34.970360    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8zf7t_6ee3e5a6-186c-46a2-aef3-362ea08f9b30/kube-rbac-proxy/0.log"
Jul 21 17:28:36.171440 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:36.171402    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8gvpd_85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15/dns-node-resolver/0.log"
Jul 21 17:28:39.879084 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:39.878964    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54dc988fb9-bqp2l"]
Jul 21 17:28:46.647074 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:46.647040    2562 generic.go:356] "Generic (PLEG): container finished" podID="813ad497-8605-4f33-b614-32c185d701fc" containerID="a300b3d58a8dc042e9a5998d7d1f5d36b47c4b078e74db0a428fa9858614ec63" exitCode=0
Jul 21 17:28:46.647451 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:46.647110    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n" event={"ID":"813ad497-8605-4f33-b614-32c185d701fc","Type":"ContainerDied","Data":"a300b3d58a8dc042e9a5998d7d1f5d36b47c4b078e74db0a428fa9858614ec63"}
Jul 21 17:28:46.647451 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:46.647412    2562 scope.go:117] "RemoveContainer" containerID="a300b3d58a8dc042e9a5998d7d1f5d36b47c4b078e74db0a428fa9858614ec63"
Jul 21 17:28:47.653905 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:28:47.653876    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-56c8n" event={"ID":"813ad497-8605-4f33-b614-32c185d701fc","Type":"ContainerStarted","Data":"81d6889839f933402c351aeb262e4785fdce2fc7e65eab7cec03ac4a3c832df3"}
Jul 21 17:29:04.907745 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:04.907678    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54dc988fb9-bqp2l" podUID="f22d5b11-93c8-4397-992c-441324d1a24f" containerName="console" containerID="cri-o://bfd24e2146556107311d761c140f0854b0f9060421deac050038a378d150ad21" gracePeriod=15
Jul 21 17:29:05.157008 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.156985    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:29:05.271506 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.271414    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f22d5b11-93c8-4397-992c-441324d1a24f-console-serving-cert\") pod \"f22d5b11-93c8-4397-992c-441324d1a24f\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") "
Jul 21 17:29:05.271506 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.271471    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f22d5b11-93c8-4397-992c-441324d1a24f-console-oauth-config\") pod \"f22d5b11-93c8-4397-992c-441324d1a24f\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") "
Jul 21 17:29:05.271735 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.271518    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-console-config\") pod \"f22d5b11-93c8-4397-992c-441324d1a24f\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") "
Jul 21 17:29:05.271735 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.271538    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-service-ca\") pod \"f22d5b11-93c8-4397-992c-441324d1a24f\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") "
Jul 21 17:29:05.271735 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.271575    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-oauth-serving-cert\") pod \"f22d5b11-93c8-4397-992c-441324d1a24f\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") "
Jul 21 17:29:05.271735 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.271615    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8vkc\" (UniqueName: \"kubernetes.io/projected/f22d5b11-93c8-4397-992c-441324d1a24f-kube-api-access-s8vkc\") pod \"f22d5b11-93c8-4397-992c-441324d1a24f\" (UID: \"f22d5b11-93c8-4397-992c-441324d1a24f\") "
Jul 21 17:29:05.271955 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.271922    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-service-ca" (OuterVolumeSpecName: "service-ca") pod "f22d5b11-93c8-4397-992c-441324d1a24f" (UID: "f22d5b11-93c8-4397-992c-441324d1a24f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:29:05.272004 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.271977    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-console-config" (OuterVolumeSpecName: "console-config") pod "f22d5b11-93c8-4397-992c-441324d1a24f" (UID: "f22d5b11-93c8-4397-992c-441324d1a24f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:29:05.272089 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.272012    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f22d5b11-93c8-4397-992c-441324d1a24f" (UID: "f22d5b11-93c8-4397-992c-441324d1a24f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:29:05.273911 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.273886    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f22d5b11-93c8-4397-992c-441324d1a24f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f22d5b11-93c8-4397-992c-441324d1a24f" (UID: "f22d5b11-93c8-4397-992c-441324d1a24f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:05.274023 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.273916    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f22d5b11-93c8-4397-992c-441324d1a24f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f22d5b11-93c8-4397-992c-441324d1a24f" (UID: "f22d5b11-93c8-4397-992c-441324d1a24f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:05.274023 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.273949    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22d5b11-93c8-4397-992c-441324d1a24f-kube-api-access-s8vkc" (OuterVolumeSpecName: "kube-api-access-s8vkc") pod "f22d5b11-93c8-4397-992c-441324d1a24f" (UID: "f22d5b11-93c8-4397-992c-441324d1a24f"). InnerVolumeSpecName "kube-api-access-s8vkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:29:05.372722 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.372688    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f22d5b11-93c8-4397-992c-441324d1a24f-console-oauth-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:29:05.372722 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.372717    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-console-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:29:05.372722 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.372726    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-service-ca\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:29:05.372955 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.372735    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f22d5b11-93c8-4397-992c-441324d1a24f-oauth-serving-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:29:05.372955 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.372743    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8vkc\" (UniqueName: \"kubernetes.io/projected/f22d5b11-93c8-4397-992c-441324d1a24f-kube-api-access-s8vkc\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:29:05.372955 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.372752    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f22d5b11-93c8-4397-992c-441324d1a24f-console-serving-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:29:05.722447 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.722416    2562 generic.go:356] "Generic (PLEG): container finished" podID="f22d5b11-93c8-4397-992c-441324d1a24f" containerID="bfd24e2146556107311d761c140f0854b0f9060421deac050038a378d150ad21" exitCode=0
Jul 21 17:29:05.722589 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.722459    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54dc988fb9-bqp2l"
Jul 21 17:29:05.722589 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.722461    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54dc988fb9-bqp2l" event={"ID":"f22d5b11-93c8-4397-992c-441324d1a24f","Type":"ContainerDied","Data":"bfd24e2146556107311d761c140f0854b0f9060421deac050038a378d150ad21"}
Jul 21 17:29:05.722589 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.722503    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54dc988fb9-bqp2l" event={"ID":"f22d5b11-93c8-4397-992c-441324d1a24f","Type":"ContainerDied","Data":"02d8c4e2f5c66569063c54686d0324bde4d9f2aeba4b19fb80c784c2774f42ff"}
Jul 21 17:29:05.722589 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.722522    2562 scope.go:117] "RemoveContainer" containerID="bfd24e2146556107311d761c140f0854b0f9060421deac050038a378d150ad21"
Jul 21 17:29:05.733050 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.732166    2562 scope.go:117] "RemoveContainer" containerID="bfd24e2146556107311d761c140f0854b0f9060421deac050038a378d150ad21"
Jul 21 17:29:05.733620 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:29:05.733598    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd24e2146556107311d761c140f0854b0f9060421deac050038a378d150ad21\": container with ID starting with bfd24e2146556107311d761c140f0854b0f9060421deac050038a378d150ad21 not found: ID does not exist" containerID="bfd24e2146556107311d761c140f0854b0f9060421deac050038a378d150ad21"
Jul 21 17:29:05.733691 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.733628    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd24e2146556107311d761c140f0854b0f9060421deac050038a378d150ad21"} err="failed to get container status \"bfd24e2146556107311d761c140f0854b0f9060421deac050038a378d150ad21\": rpc error: code = NotFound desc = could not find container \"bfd24e2146556107311d761c140f0854b0f9060421deac050038a378d150ad21\": container with ID starting with bfd24e2146556107311d761c140f0854b0f9060421deac050038a378d150ad21 not found: ID does not exist"
Jul 21 17:29:05.742947 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.742917    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54dc988fb9-bqp2l"]
Jul 21 17:29:05.746543 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:05.746522    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54dc988fb9-bqp2l"]
Jul 21 17:29:06.920928 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:29:06.920897    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f22d5b11-93c8-4397-992c-441324d1a24f" path="/var/lib/kubelet/pods/f22d5b11-93c8-4397-992c-441324d1a24f/volumes"
Jul 21 17:30:29.169078 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.169041    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt"]
Jul 21 17:30:29.169507 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.169331    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f22d5b11-93c8-4397-992c-441324d1a24f" containerName="console"
Jul 21 17:30:29.169507 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.169341    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f22d5b11-93c8-4397-992c-441324d1a24f" containerName="console"
Jul 21 17:30:29.169507 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.169387    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f22d5b11-93c8-4397-992c-441324d1a24f" containerName="console"
Jul 21 17:30:29.172413 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.172398    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt"
Jul 21 17:30:29.178802 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.178782    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 21 17:30:29.179238 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.179221    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 21 17:30:29.179292 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.179221    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fzv6p\""
Jul 21 17:30:29.199574 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.199533    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt"]
Jul 21 17:30:29.226271 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.226230    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt\" (UID: \"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt"
Jul 21 17:30:29.226271 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.226273    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt\" (UID: \"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt"
Jul 21 17:30:29.226499 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.226351    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9mc5\" (UniqueName: \"kubernetes.io/projected/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-kube-api-access-n9mc5\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt\" (UID: \"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt"
Jul 21 17:30:29.327627 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.327574    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt\" (UID: \"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt"
Jul 21 17:30:29.327809 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.327651    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9mc5\" (UniqueName: \"kubernetes.io/projected/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-kube-api-access-n9mc5\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt\" (UID: \"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt"
Jul 21 17:30:29.327809 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.327703    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt\" (UID: \"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt"
Jul 21 17:30:29.327985 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.327963    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt\" (UID: \"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt"
Jul 21 17:30:29.328021 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.327984    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt\" (UID: \"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt"
Jul 21 17:30:29.338028 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.337995    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9mc5\" (UniqueName: \"kubernetes.io/projected/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-kube-api-access-n9mc5\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt\" (UID: \"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt"
Jul 21 17:30:29.481089 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.480995    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt"
Jul 21 17:30:29.611450 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:29.611425    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt"]
Jul 21 17:30:29.613435 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:30:29.613412    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac689c8_d27c_45f5_a4ac_d6ff0abc5c7e.slice/crio-493865fee4dbf899371a52e141360ee8b3bd586dee72ad19b697ae301c800f26 WatchSource:0}: Error finding container 493865fee4dbf899371a52e141360ee8b3bd586dee72ad19b697ae301c800f26: Status 404 returned error can't find the container with id 493865fee4dbf899371a52e141360ee8b3bd586dee72ad19b697ae301c800f26
Jul 21 17:30:30.032117 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:30.032085    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt" event={"ID":"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e","Type":"ContainerStarted","Data":"493865fee4dbf899371a52e141360ee8b3bd586dee72ad19b697ae301c800f26"}
Jul 21 17:30:35.055397 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:35.055363    2562 generic.go:356] "Generic (PLEG): container finished" podID="bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e" containerID="a800def6891a06967fff52caf62afeb9762f0de3c3bb4d20bede15abab4018e3" exitCode=0
Jul 21 17:30:35.055758 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:35.055451    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt" event={"ID":"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e","Type":"ContainerDied","Data":"a800def6891a06967fff52caf62afeb9762f0de3c3bb4d20bede15abab4018e3"}
Jul 21 17:30:38.071413 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:38.071383    2562 generic.go:356] "Generic (PLEG): container finished" podID="bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e" containerID="8a8d4b40f3a0660b67a3fa7435d6bd969b9a40c7f1ecb96956f0f85d778f20e5" exitCode=0
Jul 21 17:30:38.071810 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:38.071445    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt" event={"ID":"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e","Type":"ContainerDied","Data":"8a8d4b40f3a0660b67a3fa7435d6bd969b9a40c7f1ecb96956f0f85d778f20e5"}
Jul 21 17:30:45.108121 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:45.108019    2562 generic.go:356] "Generic (PLEG): container finished" podID="bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e" containerID="42ea0c64122668ec1bdea7f511a6cf0534b57656cd704b7ffb30d2889b379acf" exitCode=0
Jul 21 17:30:45.108471 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:45.108113    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt" event={"ID":"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e","Type":"ContainerDied","Data":"42ea0c64122668ec1bdea7f511a6cf0534b57656cd704b7ffb30d2889b379acf"}
Jul 21 17:30:46.234341 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:46.234318    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt"
Jul 21 17:30:46.378596 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:46.378492    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-util\") pod \"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e\" (UID: \"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e\") "
Jul 21 17:30:46.378596 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:46.378563    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-bundle\") pod \"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e\" (UID: \"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e\") "
Jul 21 17:30:46.378820 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:46.378648    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9mc5\" (UniqueName: \"kubernetes.io/projected/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-kube-api-access-n9mc5\") pod \"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e\" (UID: \"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e\") "
Jul 21 17:30:46.379240 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:46.379204    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-bundle" (OuterVolumeSpecName: "bundle") pod "bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e" (UID: "bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:30:46.380834 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:46.380815    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-kube-api-access-n9mc5" (OuterVolumeSpecName: "kube-api-access-n9mc5") pod "bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e" (UID: "bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e"). InnerVolumeSpecName "kube-api-access-n9mc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:30:46.382542 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:46.382518    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-util" (OuterVolumeSpecName: "util") pod "bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e" (UID: "bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:30:46.479665 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:46.479621    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-bundle\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:30:46.479665 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:46.479662    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9mc5\" (UniqueName: \"kubernetes.io/projected/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-kube-api-access-n9mc5\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:30:46.479665 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:46.479673    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e-util\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:30:47.118371 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:47.118337    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt" event={"ID":"bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e","Type":"ContainerDied","Data":"493865fee4dbf899371a52e141360ee8b3bd586dee72ad19b697ae301c800f26"}
Jul 21 17:30:47.118371 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:47.118359    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xh7bt"
Jul 21 17:30:47.118561 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:47.118381    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="493865fee4dbf899371a52e141360ee8b3bd586dee72ad19b697ae301c800f26"
Jul 21 17:30:48.796622 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:48.796592    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 17:30:48.798632 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:48.798607    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 17:30:48.808383 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:48.808361    2562 kubelet.go:1648] "Image garbage collection succeeded"
Jul 21 17:30:56.283996 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.283963    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-crnjp"]
Jul 21 17:30:56.335819 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.284263    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e" containerName="pull"
Jul 21 17:30:56.335819 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.284274    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e" containerName="pull"
Jul 21 17:30:56.335819 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.284287    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e" containerName="extract"
Jul 21 17:30:56.335819 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.284292    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e" containerName="extract"
Jul 21 17:30:56.335819 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.284306    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e" containerName="util"
Jul 21 17:30:56.335819 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.284315    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e" containerName="util"
Jul 21 17:30:56.335819 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.284376    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac689c8-d27c-45f5-a4ac-d6ff0abc5c7e" containerName="extract"
Jul 21 17:30:56.345003 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.344968    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-crnjp"]
Jul 21 17:30:56.345159 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.345138    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-crnjp"
Jul 21 17:30:56.348459 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.348427    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 21 17:30:56.348709 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.348429    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 21 17:30:56.348839 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.348822    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 21 17:30:56.349674 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.349656    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 21 17:30:56.349794 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.349686    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 21 17:30:56.349794 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.349699    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-c6sxk\""
Jul 21 17:30:56.454786 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.454761    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8752201d-bcab-4b00-9f8d-0686609f0f4d-cabundle0\") pod \"keda-operator-77f648d694-crnjp\" (UID: \"8752201d-bcab-4b00-9f8d-0686609f0f4d\") " pod="openshift-keda/keda-operator-77f648d694-crnjp"
Jul 21 17:30:56.454942 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.454792    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8752201d-bcab-4b00-9f8d-0686609f0f4d-certificates\") pod \"keda-operator-77f648d694-crnjp\" (UID: \"8752201d-bcab-4b00-9f8d-0686609f0f4d\") " pod="openshift-keda/keda-operator-77f648d694-crnjp"
Jul 21 17:30:56.454942 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.454826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfpgf\" (UniqueName: \"kubernetes.io/projected/8752201d-bcab-4b00-9f8d-0686609f0f4d-kube-api-access-mfpgf\") pod \"keda-operator-77f648d694-crnjp\" (UID: \"8752201d-bcab-4b00-9f8d-0686609f0f4d\") " pod="openshift-keda/keda-operator-77f648d694-crnjp"
Jul 21 17:30:56.498751 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.498721    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-726jt"]
Jul 21 17:30:56.514785 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.514755    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-726jt"]
Jul 21 17:30:56.514909 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.514894    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt"
Jul 21 17:30:56.517479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.517460    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 21 17:30:56.555948 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.555890    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8752201d-bcab-4b00-9f8d-0686609f0f4d-cabundle0\") pod \"keda-operator-77f648d694-crnjp\" (UID: \"8752201d-bcab-4b00-9f8d-0686609f0f4d\") " pod="openshift-keda/keda-operator-77f648d694-crnjp"
Jul 21 17:30:56.555948 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.555919    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8752201d-bcab-4b00-9f8d-0686609f0f4d-certificates\") pod \"keda-operator-77f648d694-crnjp\" (UID: \"8752201d-bcab-4b00-9f8d-0686609f0f4d\") " pod="openshift-keda/keda-operator-77f648d694-crnjp"
Jul 21 17:30:56.556127 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.555952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfpgf\" (UniqueName: \"kubernetes.io/projected/8752201d-bcab-4b00-9f8d-0686609f0f4d-kube-api-access-mfpgf\") pod \"keda-operator-77f648d694-crnjp\" (UID: \"8752201d-bcab-4b00-9f8d-0686609f0f4d\") " pod="openshift-keda/keda-operator-77f648d694-crnjp"
Jul 21 17:30:56.556127 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:30:56.556084    2562 secret.go:281] references non-existent secret key: ca.crt
Jul 21 17:30:56.556127 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:30:56.556105    2562 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 21 17:30:56.556127 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:30:56.556116    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-crnjp: references non-existent secret key: ca.crt
Jul 21 17:30:56.556267 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:30:56.556175    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8752201d-bcab-4b00-9f8d-0686609f0f4d-certificates podName:8752201d-bcab-4b00-9f8d-0686609f0f4d nodeName:}" failed. No retries permitted until 2026-07-21 17:30:57.056154728 +0000 UTC m=+308.627776476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/8752201d-bcab-4b00-9f8d-0686609f0f4d-certificates") pod "keda-operator-77f648d694-crnjp" (UID: "8752201d-bcab-4b00-9f8d-0686609f0f4d") : references non-existent secret key: ca.crt
Jul 21 17:30:56.556492 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.556475    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8752201d-bcab-4b00-9f8d-0686609f0f4d-cabundle0\") pod \"keda-operator-77f648d694-crnjp\" (UID: \"8752201d-bcab-4b00-9f8d-0686609f0f4d\") " pod="openshift-keda/keda-operator-77f648d694-crnjp"
Jul 21 17:30:56.569734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.569713    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfpgf\" (UniqueName: \"kubernetes.io/projected/8752201d-bcab-4b00-9f8d-0686609f0f4d-kube-api-access-mfpgf\") pod \"keda-operator-77f648d694-crnjp\" (UID: \"8752201d-bcab-4b00-9f8d-0686609f0f4d\") " pod="openshift-keda/keda-operator-77f648d694-crnjp"
Jul 21 17:30:56.656770 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.656734    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfnwb\" (UniqueName: \"kubernetes.io/projected/4c936b17-94b8-406c-89a5-ebff4f058436-kube-api-access-rfnwb\") pod \"keda-metrics-apiserver-55557df8c-726jt\" (UID: \"4c936b17-94b8-406c-89a5-ebff4f058436\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt"
Jul 21 17:30:56.656770 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.656774    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4c936b17-94b8-406c-89a5-ebff4f058436-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-726jt\" (UID: \"4c936b17-94b8-406c-89a5-ebff4f058436\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt"
Jul 21 17:30:56.656991 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.656807    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4c936b17-94b8-406c-89a5-ebff4f058436-certificates\") pod \"keda-metrics-apiserver-55557df8c-726jt\" (UID: \"4c936b17-94b8-406c-89a5-ebff4f058436\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt"
Jul 21 17:30:56.758207 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.758169    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4c936b17-94b8-406c-89a5-ebff4f058436-certificates\") pod \"keda-metrics-apiserver-55557df8c-726jt\" (UID: \"4c936b17-94b8-406c-89a5-ebff4f058436\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt"
Jul 21 17:30:56.758351 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.758267    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfnwb\" (UniqueName: \"kubernetes.io/projected/4c936b17-94b8-406c-89a5-ebff4f058436-kube-api-access-rfnwb\") pod \"keda-metrics-apiserver-55557df8c-726jt\" (UID: \"4c936b17-94b8-406c-89a5-ebff4f058436\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt"
Jul 21 17:30:56.758351 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.758301    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4c936b17-94b8-406c-89a5-ebff4f058436-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-726jt\" (UID: \"4c936b17-94b8-406c-89a5-ebff4f058436\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt"
Jul 21 17:30:56.758351 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:30:56.758330    2562 secret.go:281] references non-existent secret key: tls.crt
Jul 21 17:30:56.758351 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:30:56.758348    2562 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 21 17:30:56.758494 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:30:56.758368    2562 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 21 17:30:56.758494 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:30:56.758389    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-726jt: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 21 17:30:56.758494 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:30:56.758456    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c936b17-94b8-406c-89a5-ebff4f058436-certificates podName:4c936b17-94b8-406c-89a5-ebff4f058436 nodeName:}" failed. No retries permitted until 2026-07-21 17:30:57.258438628 +0000 UTC m=+308.830060367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4c936b17-94b8-406c-89a5-ebff4f058436-certificates") pod "keda-metrics-apiserver-55557df8c-726jt" (UID: "4c936b17-94b8-406c-89a5-ebff4f058436") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 21 17:30:56.758733 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.758716    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4c936b17-94b8-406c-89a5-ebff4f058436-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-726jt\" (UID: \"4c936b17-94b8-406c-89a5-ebff4f058436\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt"
Jul 21 17:30:56.768597 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:56.768571    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfnwb\" (UniqueName: \"kubernetes.io/projected/4c936b17-94b8-406c-89a5-ebff4f058436-kube-api-access-rfnwb\") pod \"keda-metrics-apiserver-55557df8c-726jt\" (UID: \"4c936b17-94b8-406c-89a5-ebff4f058436\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt"
Jul 21 17:30:57.060944 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:57.060904    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8752201d-bcab-4b00-9f8d-0686609f0f4d-certificates\") pod \"keda-operator-77f648d694-crnjp\" (UID: \"8752201d-bcab-4b00-9f8d-0686609f0f4d\") " pod="openshift-keda/keda-operator-77f648d694-crnjp"
Jul 21 17:30:57.063416 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:57.063393    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8752201d-bcab-4b00-9f8d-0686609f0f4d-certificates\") pod \"keda-operator-77f648d694-crnjp\" (UID: \"8752201d-bcab-4b00-9f8d-0686609f0f4d\") " pod="openshift-keda/keda-operator-77f648d694-crnjp"
Jul 21 17:30:57.255041 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:57.255007    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-crnjp"
Jul 21 17:30:57.262344 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:57.262322    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4c936b17-94b8-406c-89a5-ebff4f058436-certificates\") pod \"keda-metrics-apiserver-55557df8c-726jt\" (UID: \"4c936b17-94b8-406c-89a5-ebff4f058436\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt"
Jul 21 17:30:57.264993 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:57.264970    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4c936b17-94b8-406c-89a5-ebff4f058436-certificates\") pod \"keda-metrics-apiserver-55557df8c-726jt\" (UID: \"4c936b17-94b8-406c-89a5-ebff4f058436\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt"
Jul 21 17:30:57.405815 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:57.405790    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-crnjp"]
Jul 21 17:30:57.408424 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:30:57.408389    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8752201d_bcab_4b00_9f8d_0686609f0f4d.slice/crio-ee71fdd8523b9a079631978fb475e056a864a31528ae476d3d24b933300b0d68 WatchSource:0}: Error finding container ee71fdd8523b9a079631978fb475e056a864a31528ae476d3d24b933300b0d68: Status 404 returned error can't find the container with id ee71fdd8523b9a079631978fb475e056a864a31528ae476d3d24b933300b0d68
Jul 21 17:30:57.412098 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:57.410200    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 17:30:57.425272 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:57.425247    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt"
Jul 21 17:30:57.582194 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:57.582171    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-726jt"]
Jul 21 17:30:57.583778 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:30:57.583750    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c936b17_94b8_406c_89a5_ebff4f058436.slice/crio-8a786c63721c1015ba071b1d324eefeb1d19621f9dbc968ba762f46e5504b012 WatchSource:0}: Error finding container 8a786c63721c1015ba071b1d324eefeb1d19621f9dbc968ba762f46e5504b012: Status 404 returned error can't find the container with id 8a786c63721c1015ba071b1d324eefeb1d19621f9dbc968ba762f46e5504b012
Jul 21 17:30:58.158579 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:58.158545    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt" event={"ID":"4c936b17-94b8-406c-89a5-ebff4f058436","Type":"ContainerStarted","Data":"8a786c63721c1015ba071b1d324eefeb1d19621f9dbc968ba762f46e5504b012"}
Jul 21 17:30:58.159956 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:30:58.159919    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-crnjp" event={"ID":"8752201d-bcab-4b00-9f8d-0686609f0f4d","Type":"ContainerStarted","Data":"ee71fdd8523b9a079631978fb475e056a864a31528ae476d3d24b933300b0d68"}
Jul 21 17:31:02.177977 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:31:02.177933    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt" event={"ID":"4c936b17-94b8-406c-89a5-ebff4f058436","Type":"ContainerStarted","Data":"707471c85c752f9778aca5cde385b1772a85f8b6fb4d62993734db3c06c5a4b2"}
Jul 21 17:31:02.178494 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:31:02.178107    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt"
Jul 21 17:31:02.179452 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:31:02.179428    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-crnjp" event={"ID":"8752201d-bcab-4b00-9f8d-0686609f0f4d","Type":"ContainerStarted","Data":"8bc3dd41685e8d464b8f43fcb0f66b21182300e58dc0be4019eccff8975be05f"}
Jul 21 17:31:02.179577 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:31:02.179557    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-crnjp"
Jul 21 17:31:02.196131 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:31:02.196083    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt" podStartSLOduration=2.491085568 podStartE2EDuration="6.19605281s" totalImagesPullingTime="3.704967242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:31:02.194900092 +0000 UTC m=+313.766521849" watchObservedRunningTime="2026-07-21 17:31:02.19605281 +0000 UTC m=+313.767674543"
Jul 21 17:31:02.212462 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:31:02.212416    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-crnjp" podStartSLOduration=2.329601302 podStartE2EDuration="6.212400947s" totalImagesPullingTime="3.882799645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:31:02.211450879 +0000 UTC m=+313.783072634" watchObservedRunningTime="2026-07-21 17:31:02.212400947 +0000 UTC m=+313.784022701"
Jul 21 17:31:13.187103 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:31:13.187056    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-726jt"
Jul 21 17:31:23.185919 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:31:23.185882    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-crnjp"
Jul 21 17:32:13.385602 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:13.385561    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-jvmmm"]
Jul 21 17:32:13.388857 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:13.388829    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-jvmmm"
Jul 21 17:32:13.392369 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:13.392347    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 21 17:32:13.393519 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:13.393503    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 21 17:32:13.393519 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:13.393517    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-f4924\""
Jul 21 17:32:13.393681 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:13.393508    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 21 17:32:13.402517 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:13.402497    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-jvmmm"]
Jul 21 17:32:13.542877 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:13.542840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd58785d-8d50-4068-a534-17aefa8fafd5-cert\") pod \"llmisvc-controller-manager-654c557856-jvmmm\" (UID: \"dd58785d-8d50-4068-a534-17aefa8fafd5\") " pod="kserve/llmisvc-controller-manager-654c557856-jvmmm"
Jul 21 17:32:13.543100 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:13.542923    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmft9\" (UniqueName: \"kubernetes.io/projected/dd58785d-8d50-4068-a534-17aefa8fafd5-kube-api-access-vmft9\") pod \"llmisvc-controller-manager-654c557856-jvmmm\" (UID: \"dd58785d-8d50-4068-a534-17aefa8fafd5\") " pod="kserve/llmisvc-controller-manager-654c557856-jvmmm"
Jul 21 17:32:13.643671 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:13.643581    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd58785d-8d50-4068-a534-17aefa8fafd5-cert\") pod \"llmisvc-controller-manager-654c557856-jvmmm\" (UID: \"dd58785d-8d50-4068-a534-17aefa8fafd5\") " pod="kserve/llmisvc-controller-manager-654c557856-jvmmm"
Jul 21 17:32:13.643671 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:13.643653    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmft9\" (UniqueName: \"kubernetes.io/projected/dd58785d-8d50-4068-a534-17aefa8fafd5-kube-api-access-vmft9\") pod \"llmisvc-controller-manager-654c557856-jvmmm\" (UID: \"dd58785d-8d50-4068-a534-17aefa8fafd5\") " pod="kserve/llmisvc-controller-manager-654c557856-jvmmm"
Jul 21 17:32:13.646341 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:13.646312    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd58785d-8d50-4068-a534-17aefa8fafd5-cert\") pod \"llmisvc-controller-manager-654c557856-jvmmm\" (UID: \"dd58785d-8d50-4068-a534-17aefa8fafd5\") " pod="kserve/llmisvc-controller-manager-654c557856-jvmmm"
Jul 21 17:32:13.653578 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:13.653553    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmft9\" (UniqueName: \"kubernetes.io/projected/dd58785d-8d50-4068-a534-17aefa8fafd5-kube-api-access-vmft9\") pod \"llmisvc-controller-manager-654c557856-jvmmm\" (UID: \"dd58785d-8d50-4068-a534-17aefa8fafd5\") " pod="kserve/llmisvc-controller-manager-654c557856-jvmmm"
Jul 21 17:32:13.699110 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:13.699060    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-jvmmm"
Jul 21 17:32:13.826832 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:13.826806    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-jvmmm"]
Jul 21 17:32:13.829479 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:32:13.829449    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddd58785d_8d50_4068_a534_17aefa8fafd5.slice/crio-eb67a066472a4c9456a3dbcbdf366112c8f1ce20d6ebe2491a5c3a24c653f7e6 WatchSource:0}: Error finding container eb67a066472a4c9456a3dbcbdf366112c8f1ce20d6ebe2491a5c3a24c653f7e6: Status 404 returned error can't find the container with id eb67a066472a4c9456a3dbcbdf366112c8f1ce20d6ebe2491a5c3a24c653f7e6
Jul 21 17:32:14.453849 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:14.453808    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-jvmmm" event={"ID":"dd58785d-8d50-4068-a534-17aefa8fafd5","Type":"ContainerStarted","Data":"eb67a066472a4c9456a3dbcbdf366112c8f1ce20d6ebe2491a5c3a24c653f7e6"}
Jul 21 17:32:20.480153 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:20.480111    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-jvmmm" event={"ID":"dd58785d-8d50-4068-a534-17aefa8fafd5","Type":"ContainerStarted","Data":"ec89c98a5eb7d34677966c59c40e01094451dff052998c0804de468df73d72ff"}
Jul 21 17:32:20.480537 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:20.480229    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-jvmmm"
Jul 21 17:32:20.500551 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:20.500500    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-jvmmm" podStartSLOduration=1.65322554 podStartE2EDuration="7.500484227s" totalImagesPullingTime="5.847258687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:32:20.498628927 +0000 UTC m=+392.070250729" watchObservedRunningTime="2026-07-21 17:32:20.500484227 +0000 UTC m=+392.072105982"
Jul 21 17:32:51.487766 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:32:51.487737    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-jvmmm"
Jul 21 17:33:32.805799 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.805761    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-6cvvv"]
Jul 21 17:33:32.808040 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.808022    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-6cvvv"
Jul 21 17:33:32.810987 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.810959    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 21 17:33:32.811101 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.810972    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-87wzx\""
Jul 21 17:33:32.820673 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.820653    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-lnjkt"]
Jul 21 17:33:32.822532 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.822514    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-6cvvv"]
Jul 21 17:33:32.822671 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.822654    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-lnjkt"
Jul 21 17:33:32.825640 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.825620    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 21 17:33:32.826280 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.826263    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-4fdgq\""
Jul 21 17:33:32.835633 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.835612    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-lnjkt"]
Jul 21 17:33:32.886053 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.886027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65b4x\" (UniqueName: \"kubernetes.io/projected/13722b77-a492-41cf-a54b-ca8845acc44e-kube-api-access-65b4x\") pod \"model-serving-api-fd65d7d6-6cvvv\" (UID: \"13722b77-a492-41cf-a54b-ca8845acc44e\") " pod="kserve/model-serving-api-fd65d7d6-6cvvv"
Jul 21 17:33:32.886175 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.886102    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13722b77-a492-41cf-a54b-ca8845acc44e-tls-certs\") pod \"model-serving-api-fd65d7d6-6cvvv\" (UID: \"13722b77-a492-41cf-a54b-ca8845acc44e\") " pod="kserve/model-serving-api-fd65d7d6-6cvvv"
Jul 21 17:33:32.987266 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.987238    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj588\" (UniqueName: \"kubernetes.io/projected/beba3329-ac69-4168-a5cf-7e171d9f8da5-kube-api-access-pj588\") pod \"odh-model-controller-84b54b7f97-lnjkt\" (UID: \"beba3329-ac69-4168-a5cf-7e171d9f8da5\") " pod="kserve/odh-model-controller-84b54b7f97-lnjkt"
Jul 21 17:33:32.987393 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.987287    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/beba3329-ac69-4168-a5cf-7e171d9f8da5-cert\") pod \"odh-model-controller-84b54b7f97-lnjkt\" (UID: \"beba3329-ac69-4168-a5cf-7e171d9f8da5\") " pod="kserve/odh-model-controller-84b54b7f97-lnjkt"
Jul 21 17:33:32.987393 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.987319    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65b4x\" (UniqueName: \"kubernetes.io/projected/13722b77-a492-41cf-a54b-ca8845acc44e-kube-api-access-65b4x\") pod \"model-serving-api-fd65d7d6-6cvvv\" (UID: \"13722b77-a492-41cf-a54b-ca8845acc44e\") " pod="kserve/model-serving-api-fd65d7d6-6cvvv"
Jul 21 17:33:32.987393 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.987364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13722b77-a492-41cf-a54b-ca8845acc44e-tls-certs\") pod \"model-serving-api-fd65d7d6-6cvvv\" (UID: \"13722b77-a492-41cf-a54b-ca8845acc44e\") " pod="kserve/model-serving-api-fd65d7d6-6cvvv"
Jul 21 17:33:32.989836 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.989814    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13722b77-a492-41cf-a54b-ca8845acc44e-tls-certs\") pod \"model-serving-api-fd65d7d6-6cvvv\" (UID: \"13722b77-a492-41cf-a54b-ca8845acc44e\") " pod="kserve/model-serving-api-fd65d7d6-6cvvv"
Jul 21 17:33:32.998612 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:32.998587    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65b4x\" (UniqueName: \"kubernetes.io/projected/13722b77-a492-41cf-a54b-ca8845acc44e-kube-api-access-65b4x\") pod \"model-serving-api-fd65d7d6-6cvvv\" (UID: \"13722b77-a492-41cf-a54b-ca8845acc44e\") " pod="kserve/model-serving-api-fd65d7d6-6cvvv"
Jul 21 17:33:33.087740 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:33.087674    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj588\" (UniqueName: \"kubernetes.io/projected/beba3329-ac69-4168-a5cf-7e171d9f8da5-kube-api-access-pj588\") pod \"odh-model-controller-84b54b7f97-lnjkt\" (UID: \"beba3329-ac69-4168-a5cf-7e171d9f8da5\") " pod="kserve/odh-model-controller-84b54b7f97-lnjkt"
Jul 21 17:33:33.087740 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:33.087714    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/beba3329-ac69-4168-a5cf-7e171d9f8da5-cert\") pod \"odh-model-controller-84b54b7f97-lnjkt\" (UID: \"beba3329-ac69-4168-a5cf-7e171d9f8da5\") " pod="kserve/odh-model-controller-84b54b7f97-lnjkt"
Jul 21 17:33:33.090039 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:33.090021    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/beba3329-ac69-4168-a5cf-7e171d9f8da5-cert\") pod \"odh-model-controller-84b54b7f97-lnjkt\" (UID: \"beba3329-ac69-4168-a5cf-7e171d9f8da5\") " pod="kserve/odh-model-controller-84b54b7f97-lnjkt"
Jul 21 17:33:33.096559 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:33.096534    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj588\" (UniqueName: \"kubernetes.io/projected/beba3329-ac69-4168-a5cf-7e171d9f8da5-kube-api-access-pj588\") pod \"odh-model-controller-84b54b7f97-lnjkt\" (UID: \"beba3329-ac69-4168-a5cf-7e171d9f8da5\") " pod="kserve/odh-model-controller-84b54b7f97-lnjkt"
Jul 21 17:33:33.118351 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:33.118326    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-6cvvv"
Jul 21 17:33:33.132265 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:33.132242    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-lnjkt"
Jul 21 17:33:33.256577 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:33.256544    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-6cvvv"]
Jul 21 17:33:33.260408 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:33:33.260379    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13722b77_a492_41cf_a54b_ca8845acc44e.slice/crio-18e987de0d72655837f0b933078ebafe12aa12922e90ca0ab642d3bf1bbbb45c WatchSource:0}: Error finding container 18e987de0d72655837f0b933078ebafe12aa12922e90ca0ab642d3bf1bbbb45c: Status 404 returned error can't find the container with id 18e987de0d72655837f0b933078ebafe12aa12922e90ca0ab642d3bf1bbbb45c
Jul 21 17:33:33.278026 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:33.277990    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-lnjkt"]
Jul 21 17:33:33.280265 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:33:33.280238    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeba3329_ac69_4168_a5cf_7e171d9f8da5.slice/crio-fe1dd656473dfd2989a06db7cf928d71a0198bc79f7b0fceed0e8910bcb2f46a WatchSource:0}: Error finding container fe1dd656473dfd2989a06db7cf928d71a0198bc79f7b0fceed0e8910bcb2f46a: Status 404 returned error can't find the container with id fe1dd656473dfd2989a06db7cf928d71a0198bc79f7b0fceed0e8910bcb2f46a
Jul 21 17:33:33.755409 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:33.755374    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-6cvvv" event={"ID":"13722b77-a492-41cf-a54b-ca8845acc44e","Type":"ContainerStarted","Data":"18e987de0d72655837f0b933078ebafe12aa12922e90ca0ab642d3bf1bbbb45c"}
Jul 21 17:33:33.756426 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:33.756399    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-lnjkt" event={"ID":"beba3329-ac69-4168-a5cf-7e171d9f8da5","Type":"ContainerStarted","Data":"fe1dd656473dfd2989a06db7cf928d71a0198bc79f7b0fceed0e8910bcb2f46a"}
Jul 21 17:33:38.780857 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:38.780820    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-6cvvv" event={"ID":"13722b77-a492-41cf-a54b-ca8845acc44e","Type":"ContainerStarted","Data":"1d66d675d16fbcbca1d540cb8095bdb7e33d97f7d6ecd15cf0de508c6568d998"}
Jul 21 17:33:38.781327 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:38.780944    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-6cvvv"
Jul 21 17:33:38.782337 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:38.782311    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-lnjkt" event={"ID":"beba3329-ac69-4168-a5cf-7e171d9f8da5","Type":"ContainerStarted","Data":"58d5b847d657685b0e42fd07cf62a9c9a9c3a678ce43e899f626f6edc06afd23"}
Jul 21 17:33:38.782490 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:38.782466    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-lnjkt"
Jul 21 17:33:38.806096 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:38.806028    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-6cvvv" podStartSLOduration=1.929779598 podStartE2EDuration="6.80601254s" totalImagesPullingTime="4.876232942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:33:38.804885091 +0000 UTC m=+470.376506859" watchObservedRunningTime="2026-07-21 17:33:38.80601254 +0000 UTC m=+470.377634298"
Jul 21 17:33:38.825723 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:38.825686    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-lnjkt" podStartSLOduration=1.941946156 podStartE2EDuration="6.825674746s" totalImagesPullingTime="4.88372859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:33:38.825085419 +0000 UTC m=+470.396707195" watchObservedRunningTime="2026-07-21 17:33:38.825674746 +0000 UTC m=+470.397296500"
Jul 21 17:33:49.788277 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:49.788246    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-lnjkt"
Jul 21 17:33:49.790387 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:33:49.790365    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-6cvvv"
Jul 21 17:35:29.006401 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.006367    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-z25x7"]
Jul 21 17:35:29.009733 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.009708    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-z25x7"
Jul 21 17:35:29.012753 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.012727    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 21 17:35:29.012900 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.012730    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-kwgsb\""
Jul 21 17:35:29.016538 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.016513    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 21 17:35:29.018994 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.018970    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-z25x7"]
Jul 21 17:35:29.130581 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.130546    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzlz6\" (UniqueName: \"kubernetes.io/projected/35e0b1cb-3151-4661-84df-9b281ea57b38-kube-api-access-wzlz6\") pod \"seaweedfs-tls-serving-67b7477c6d-z25x7\" (UID: \"35e0b1cb-3151-4661-84df-9b281ea57b38\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-z25x7"
Jul 21 17:35:29.130581 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.130585    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/35e0b1cb-3151-4661-84df-9b281ea57b38-data\") pod \"seaweedfs-tls-serving-67b7477c6d-z25x7\" (UID: \"35e0b1cb-3151-4661-84df-9b281ea57b38\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-z25x7"
Jul 21 17:35:29.130779 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.130622    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/35e0b1cb-3151-4661-84df-9b281ea57b38-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-z25x7\" (UID: \"35e0b1cb-3151-4661-84df-9b281ea57b38\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-z25x7"
Jul 21 17:35:29.231420 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.231380    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzlz6\" (UniqueName: \"kubernetes.io/projected/35e0b1cb-3151-4661-84df-9b281ea57b38-kube-api-access-wzlz6\") pod \"seaweedfs-tls-serving-67b7477c6d-z25x7\" (UID: \"35e0b1cb-3151-4661-84df-9b281ea57b38\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-z25x7"
Jul 21 17:35:29.231420 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.231420    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/35e0b1cb-3151-4661-84df-9b281ea57b38-data\") pod \"seaweedfs-tls-serving-67b7477c6d-z25x7\" (UID: \"35e0b1cb-3151-4661-84df-9b281ea57b38\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-z25x7"
Jul 21 17:35:29.231637 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.231452    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/35e0b1cb-3151-4661-84df-9b281ea57b38-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-z25x7\" (UID: \"35e0b1cb-3151-4661-84df-9b281ea57b38\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-z25x7"
Jul 21 17:35:29.231637 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:35:29.231604    2562 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Jul 21 17:35:29.231637 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:35:29.231622    2562 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-z25x7: secret "seaweedfs-tls-serving" not found
Jul 21 17:35:29.231727 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:35:29.231696    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35e0b1cb-3151-4661-84df-9b281ea57b38-seaweedfs-tls-serving podName:35e0b1cb-3151-4661-84df-9b281ea57b38 nodeName:}" failed. No retries permitted until 2026-07-21 17:35:29.73167324 +0000 UTC m=+581.303294973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/35e0b1cb-3151-4661-84df-9b281ea57b38-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-z25x7" (UID: "35e0b1cb-3151-4661-84df-9b281ea57b38") : secret "seaweedfs-tls-serving" not found
Jul 21 17:35:29.231859 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.231839    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/35e0b1cb-3151-4661-84df-9b281ea57b38-data\") pod \"seaweedfs-tls-serving-67b7477c6d-z25x7\" (UID: \"35e0b1cb-3151-4661-84df-9b281ea57b38\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-z25x7"
Jul 21 17:35:29.240214 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.240181    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzlz6\" (UniqueName: \"kubernetes.io/projected/35e0b1cb-3151-4661-84df-9b281ea57b38-kube-api-access-wzlz6\") pod \"seaweedfs-tls-serving-67b7477c6d-z25x7\" (UID: \"35e0b1cb-3151-4661-84df-9b281ea57b38\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-z25x7"
Jul 21 17:35:29.735219 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.735174    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/35e0b1cb-3151-4661-84df-9b281ea57b38-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-z25x7\" (UID: \"35e0b1cb-3151-4661-84df-9b281ea57b38\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-z25x7"
Jul 21 17:35:29.737817 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.737790    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/35e0b1cb-3151-4661-84df-9b281ea57b38-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-z25x7\" (UID: \"35e0b1cb-3151-4661-84df-9b281ea57b38\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-z25x7"
Jul 21 17:35:29.924948 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:29.924907    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-z25x7"
Jul 21 17:35:30.059980 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:30.059951    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-z25x7"]
Jul 21 17:35:30.061767 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:35:30.061742    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e0b1cb_3151_4661_84df_9b281ea57b38.slice/crio-ee2c44af959576bd27bb392aa28f4d532fe4e72a541c5fb51d487e389c0813ca WatchSource:0}: Error finding container ee2c44af959576bd27bb392aa28f4d532fe4e72a541c5fb51d487e389c0813ca: Status 404 returned error can't find the container with id ee2c44af959576bd27bb392aa28f4d532fe4e72a541c5fb51d487e389c0813ca
Jul 21 17:35:30.217055 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:30.217017    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-z25x7" event={"ID":"35e0b1cb-3151-4661-84df-9b281ea57b38","Type":"ContainerStarted","Data":"ee2c44af959576bd27bb392aa28f4d532fe4e72a541c5fb51d487e389c0813ca"}
Jul 21 17:35:33.237445 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:33.237410    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-z25x7" event={"ID":"35e0b1cb-3151-4661-84df-9b281ea57b38","Type":"ContainerStarted","Data":"c3882ee5e568f87177c968c80eb2414a1ce21d2081988bd50254aded30056aaf"}
Jul 21 17:35:33.256162 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:33.256115    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-z25x7" podStartSLOduration=2.180059117 podStartE2EDuration="5.256098309s" totalImagesPullingTime="3.076039192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:35:33.254813521 +0000 UTC m=+584.826435280" watchObservedRunningTime="2026-07-21 17:35:33.256098309 +0000 UTC m=+584.827720081"
Jul 21 17:35:48.826305 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:48.826275    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 17:35:48.827619 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:48.827599    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 17:35:57.522794 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.522755    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"]
Jul 21 17:35:57.526844 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.526826    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:57.529690 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.529649    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 21 17:35:57.529690 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.529685    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 17:35:57.529874 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.529734    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 17:35:57.529874 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.529753    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 17:35:57.530154 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.530137    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 17:35:57.530154 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.530150    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 21 17:35:57.530248 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.530150    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 17:35:57.538184 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.538163    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"]
Jul 21 17:35:57.571622 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.571591    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c02beef6-cae3-4f87-a330-64b650e11d99-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:57.571749 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.571624    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c02beef6-cae3-4f87-a330-64b650e11d99-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:57.571749 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.571684    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c02beef6-cae3-4f87-a330-64b650e11d99-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:57.571862 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.571748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-757tw\" (UniqueName: \"kubernetes.io/projected/c02beef6-cae3-4f87-a330-64b650e11d99-kube-api-access-757tw\") pod \"isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:57.571862 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.571812    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c02beef6-cae3-4f87-a330-64b650e11d99-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:57.673122 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.673085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c02beef6-cae3-4f87-a330-64b650e11d99-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:57.673275 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.673157    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-757tw\" (UniqueName: \"kubernetes.io/projected/c02beef6-cae3-4f87-a330-64b650e11d99-kube-api-access-757tw\") pod \"isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:57.673275 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.673204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c02beef6-cae3-4f87-a330-64b650e11d99-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:57.673275 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.673235    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c02beef6-cae3-4f87-a330-64b650e11d99-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:57.673275 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.673260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c02beef6-cae3-4f87-a330-64b650e11d99-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:57.673483 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:35:57.673325    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 21 17:35:57.673483 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:35:57.673393    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02beef6-cae3-4f87-a330-64b650e11d99-proxy-tls podName:c02beef6-cae3-4f87-a330-64b650e11d99 nodeName:}" failed. No retries permitted until 2026-07-21 17:35:58.173371466 +0000 UTC m=+609.744993201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c02beef6-cae3-4f87-a330-64b650e11d99-proxy-tls") pod "isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" (UID: "c02beef6-cae3-4f87-a330-64b650e11d99") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 21 17:35:57.673595 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.673550    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c02beef6-cae3-4f87-a330-64b650e11d99-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:57.673853 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.673835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c02beef6-cae3-4f87-a330-64b650e11d99-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:57.673853 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.673856    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c02beef6-cae3-4f87-a330-64b650e11d99-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:57.684547 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:57.684520    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-757tw\" (UniqueName: \"kubernetes.io/projected/c02beef6-cae3-4f87-a330-64b650e11d99-kube-api-access-757tw\") pod \"isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:58.177478 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:58.177442    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c02beef6-cae3-4f87-a330-64b650e11d99-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:58.179931 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:58.179902    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c02beef6-cae3-4f87-a330-64b650e11d99-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:58.437577 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:58.437487    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:35:58.586391 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:58.586364    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"]
Jul 21 17:35:58.588549 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:35:58.588509    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02beef6_cae3_4f87_a330_64b650e11d99.slice/crio-c576976de32d41a04682af9f655ba6baa2d3fa0293364047d43ead916d0964cf WatchSource:0}: Error finding container c576976de32d41a04682af9f655ba6baa2d3fa0293364047d43ead916d0964cf: Status 404 returned error can't find the container with id c576976de32d41a04682af9f655ba6baa2d3fa0293364047d43ead916d0964cf
Jul 21 17:35:58.590867 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:58.590851    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 17:35:59.349143 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:35:59.349113    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" event={"ID":"c02beef6-cae3-4f87-a330-64b650e11d99","Type":"ContainerStarted","Data":"c576976de32d41a04682af9f655ba6baa2d3fa0293364047d43ead916d0964cf"}
Jul 21 17:36:03.369988 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:03.369954    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" event={"ID":"c02beef6-cae3-4f87-a330-64b650e11d99","Type":"ContainerStarted","Data":"ca7e0b06b4f7bc5378498a87bcfac39474ce8c04c9315aaaa2d1f9032a6021f7"}
Jul 21 17:36:07.387863 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:07.387825    2562 generic.go:356] "Generic (PLEG): container finished" podID="c02beef6-cae3-4f87-a330-64b650e11d99" containerID="ca7e0b06b4f7bc5378498a87bcfac39474ce8c04c9315aaaa2d1f9032a6021f7" exitCode=0
Jul 21 17:36:07.388336 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:07.387898    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" event={"ID":"c02beef6-cae3-4f87-a330-64b650e11d99","Type":"ContainerDied","Data":"ca7e0b06b4f7bc5378498a87bcfac39474ce8c04c9315aaaa2d1f9032a6021f7"}
Jul 21 17:36:22.475903 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:22.475875    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" event={"ID":"c02beef6-cae3-4f87-a330-64b650e11d99","Type":"ContainerStarted","Data":"41c0524f3b79690bf6679f07362d4d06358fadaa056baf710e61d2408608bb2c"}
Jul 21 17:36:25.493165 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:25.493127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" event={"ID":"c02beef6-cae3-4f87-a330-64b650e11d99","Type":"ContainerStarted","Data":"543c5ebeeb9c4379981d4daeb1c698ba2cfe1e2cab6b4983d8d4a50862fa96ac"}
Jul 21 17:36:27.506715 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:27.506685    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" event={"ID":"c02beef6-cae3-4f87-a330-64b650e11d99","Type":"ContainerStarted","Data":"de28458817852e82bc65b78f5bdc43a27f7dcc2ce8713d0de538187f26a52037"}
Jul 21 17:36:27.507180 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:27.507003    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:36:27.507180 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:27.507086    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:36:27.507180 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:27.507124    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:36:27.508436 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:27.508391    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 21 17:36:27.509112 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:27.509091    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:36:27.538557 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:27.538502    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podStartSLOduration=3.500034674 podStartE2EDuration="30.538483778s" totalImagesPullingTime="23.137890481s" totalInitContainerRuntime="3.900558623s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:35:57 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:36:27.528112371 +0000 UTC m=+639.099734146" watchObservedRunningTime="2026-07-21 17:36:27.538483778 +0000 UTC m=+639.110105536"
Jul 21 17:36:28.511537 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:28.511492    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 21 17:36:28.511954 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:28.511893    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:36:33.515643 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:33.515613    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:36:33.516218 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:33.516191    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 21 17:36:33.516480 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:33.516453    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:36:43.516714 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:43.516678    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 21 17:36:43.517176 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:43.517149    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:36:53.516193 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:53.516140    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 21 17:36:53.516678 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:36:53.516598    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:37:03.516189 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:03.516151    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 21 17:37:03.516647 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:03.516619    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:37:13.516296 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:13.516255    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 21 17:37:13.516717 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:13.516647    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:37:23.516935 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:23.516889    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 21 17:37:23.517412 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:23.517380    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:37:33.516998 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:33.516956    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 21 17:37:33.517672 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:33.517647    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:37:43.516753 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:43.516724    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:37:43.517312 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:43.517290    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:37:52.574216 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.574183    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"]
Jul 21 17:37:52.574639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.574442    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kserve-container" containerID="cri-o://41c0524f3b79690bf6679f07362d4d06358fadaa056baf710e61d2408608bb2c" gracePeriod=30
Jul 21 17:37:52.574639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.574488    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kube-rbac-proxy" containerID="cri-o://543c5ebeeb9c4379981d4daeb1c698ba2cfe1e2cab6b4983d8d4a50862fa96ac" gracePeriod=30
Jul 21 17:37:52.574639 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.574522    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="agent" containerID="cri-o://de28458817852e82bc65b78f5bdc43a27f7dcc2ce8713d0de538187f26a52037" gracePeriod=30
Jul 21 17:37:52.604548 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.604518    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"]
Jul 21 17:37:52.608010 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.607991    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:52.610531 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.610510    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 21 17:37:52.610638 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.610556    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 21 17:37:52.621400 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.621376    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"]
Jul 21 17:37:52.675980 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.675948    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbd232cd-437f-4e8e-9412-243a16cdc2c1-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:52.676197 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.675995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:52.676197 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.676096    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:52.676197 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.676166    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbd232cd-437f-4e8e-9412-243a16cdc2c1-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:52.676321 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.676201    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjdkq\" (UniqueName: \"kubernetes.io/projected/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kube-api-access-tjdkq\") pod \"isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:52.776955 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.776917    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:52.777193 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.776963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:52.777193 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.777025    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbd232cd-437f-4e8e-9412-243a16cdc2c1-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:52.777193 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.777100    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjdkq\" (UniqueName: \"kubernetes.io/projected/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kube-api-access-tjdkq\") pod \"isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:52.777193 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.777143    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbd232cd-437f-4e8e-9412-243a16cdc2c1-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:52.777522 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.777499    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:52.777747 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.777730    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbd232cd-437f-4e8e-9412-243a16cdc2c1-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:52.777796 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.777747    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:52.779761 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.779745    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbd232cd-437f-4e8e-9412-243a16cdc2c1-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:52.786060 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.786040    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjdkq\" (UniqueName: \"kubernetes.io/projected/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kube-api-access-tjdkq\") pod \"isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:52.862979 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.862900    2562 generic.go:356] "Generic (PLEG): container finished" podID="c02beef6-cae3-4f87-a330-64b650e11d99" containerID="543c5ebeeb9c4379981d4daeb1c698ba2cfe1e2cab6b4983d8d4a50862fa96ac" exitCode=2
Jul 21 17:37:52.863140 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.862974    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" event={"ID":"c02beef6-cae3-4f87-a330-64b650e11d99","Type":"ContainerDied","Data":"543c5ebeeb9c4379981d4daeb1c698ba2cfe1e2cab6b4983d8d4a50862fa96ac"}
Jul 21 17:37:52.918052 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:52.918021    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:53.056781 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:53.056755    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"]
Jul 21 17:37:53.058884 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:37:53.058859    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd232cd_437f_4e8e_9412_243a16cdc2c1.slice/crio-70e4e479621beffde669681f25f7f4e7394cbf40311797ecc2b1afe398386878 WatchSource:0}: Error finding container 70e4e479621beffde669681f25f7f4e7394cbf40311797ecc2b1afe398386878: Status 404 returned error can't find the container with id 70e4e479621beffde669681f25f7f4e7394cbf40311797ecc2b1afe398386878
Jul 21 17:37:53.512254 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:53.512214    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 21 17:37:53.516943 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:53.516915    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:37:53.518220 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:53.518191    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 21 17:37:53.868802 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:53.868712    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" event={"ID":"bbd232cd-437f-4e8e-9412-243a16cdc2c1","Type":"ContainerStarted","Data":"fcc1b901add4c4adac90d0a4aaaf7efc7ea38c0cba7d9c3bd616d36aaf34cbb3"}
Jul 21 17:37:53.868802 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:53.868795    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" event={"ID":"bbd232cd-437f-4e8e-9412-243a16cdc2c1","Type":"ContainerStarted","Data":"70e4e479621beffde669681f25f7f4e7394cbf40311797ecc2b1afe398386878"}
Jul 21 17:37:56.884272 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:56.884240    2562 generic.go:356] "Generic (PLEG): container finished" podID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerID="fcc1b901add4c4adac90d0a4aaaf7efc7ea38c0cba7d9c3bd616d36aaf34cbb3" exitCode=0
Jul 21 17:37:56.884652 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:56.884315    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" event={"ID":"bbd232cd-437f-4e8e-9412-243a16cdc2c1","Type":"ContainerDied","Data":"fcc1b901add4c4adac90d0a4aaaf7efc7ea38c0cba7d9c3bd616d36aaf34cbb3"}
Jul 21 17:37:57.893079 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:57.893031    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" event={"ID":"bbd232cd-437f-4e8e-9412-243a16cdc2c1","Type":"ContainerStarted","Data":"221d7d86171a86a7467c688f52159ea6bac14d8214498c6e68b61382d0c5aee4"}
Jul 21 17:37:57.893547 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:57.893173    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" event={"ID":"bbd232cd-437f-4e8e-9412-243a16cdc2c1","Type":"ContainerStarted","Data":"0089cb8412e2a0833cfa2d527c82ba8282149d2c2b5f04e8461db2ccd0abd565"}
Jul 21 17:37:57.893547 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:57.893219    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" event={"ID":"bbd232cd-437f-4e8e-9412-243a16cdc2c1","Type":"ContainerStarted","Data":"82a3ef4660356731dcf09c2fce600703e4dcf3196d89106622879a35a066d6e3"}
Jul 21 17:37:57.893547 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:57.893535    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:57.893706 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:57.893670    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:57.895405 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:57.895372    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Jul 21 17:37:57.895838 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:57.895808    2562 generic.go:356] "Generic (PLEG): container finished" podID="c02beef6-cae3-4f87-a330-64b650e11d99" containerID="41c0524f3b79690bf6679f07362d4d06358fadaa056baf710e61d2408608bb2c" exitCode=0
Jul 21 17:37:57.895936 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:57.895870    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" event={"ID":"c02beef6-cae3-4f87-a330-64b650e11d99","Type":"ContainerDied","Data":"41c0524f3b79690bf6679f07362d4d06358fadaa056baf710e61d2408608bb2c"}
Jul 21 17:37:57.918600 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:57.918555    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podStartSLOduration=2.242619454 podStartE2EDuration="5.918542533s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.675923079s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:37:57.917301916 +0000 UTC m=+729.488923719" watchObservedRunningTime="2026-07-21 17:37:57.918542533 +0000 UTC m=+729.490164288"
Jul 21 17:37:58.512193 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:58.512156    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 21 17:37:58.901512 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:58.901454    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:37:58.901894 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:58.901594    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Jul 21 17:37:58.902637 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:58.902609    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:37:59.906550 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:59.906506    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Jul 21 17:37:59.906995 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:37:59.906972    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:38:03.512234 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:03.512189    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 21 17:38:03.512703 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:03.512333    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:38:03.516448 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:03.516420    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:38:03.517900 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:03.517877    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 21 17:38:04.910686 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:04.910661    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:38:04.911184 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:04.911155    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Jul 21 17:38:04.911481 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:04.911458    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:38:08.512240 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:08.512195    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 21 17:38:13.512349 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:13.512298    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 21 17:38:13.517118 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:13.517087    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:38:13.517242 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:13.517227    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:38:13.517458 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:13.517435    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 21 17:38:13.517549 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:13.517538    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:38:14.911623 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:14.911582    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Jul 21 17:38:14.912079 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:14.912037    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:38:18.511840 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:18.511801    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 21 17:38:22.732815 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:22.732787    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:38:22.845671 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:22.845575    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c02beef6-cae3-4f87-a330-64b650e11d99-proxy-tls\") pod \"c02beef6-cae3-4f87-a330-64b650e11d99\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") "
Jul 21 17:38:22.845671 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:22.845660    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-757tw\" (UniqueName: \"kubernetes.io/projected/c02beef6-cae3-4f87-a330-64b650e11d99-kube-api-access-757tw\") pod \"c02beef6-cae3-4f87-a330-64b650e11d99\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") "
Jul 21 17:38:22.845909 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:22.845685    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c02beef6-cae3-4f87-a330-64b650e11d99-cabundle-cert\") pod \"c02beef6-cae3-4f87-a330-64b650e11d99\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") "
Jul 21 17:38:22.845909 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:22.845714    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c02beef6-cae3-4f87-a330-64b650e11d99-kserve-provision-location\") pod \"c02beef6-cae3-4f87-a330-64b650e11d99\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") "
Jul 21 17:38:22.845909 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:22.845741    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c02beef6-cae3-4f87-a330-64b650e11d99-kube-rbac-proxy-sar-config\") pod \"c02beef6-cae3-4f87-a330-64b650e11d99\" (UID: \"c02beef6-cae3-4f87-a330-64b650e11d99\") "
Jul 21 17:38:22.846100 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:22.846045    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02beef6-cae3-4f87-a330-64b650e11d99-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c02beef6-cae3-4f87-a330-64b650e11d99" (UID: "c02beef6-cae3-4f87-a330-64b650e11d99"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:38:22.846210 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:22.846120    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c02beef6-cae3-4f87-a330-64b650e11d99-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c02beef6-cae3-4f87-a330-64b650e11d99" (UID: "c02beef6-cae3-4f87-a330-64b650e11d99"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:38:22.846210 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:22.846150    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c02beef6-cae3-4f87-a330-64b650e11d99-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c02beef6-cae3-4f87-a330-64b650e11d99" (UID: "c02beef6-cae3-4f87-a330-64b650e11d99"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:38:22.848001 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:22.847979    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c02beef6-cae3-4f87-a330-64b650e11d99-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c02beef6-cae3-4f87-a330-64b650e11d99" (UID: "c02beef6-cae3-4f87-a330-64b650e11d99"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:38:22.848086 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:22.848023    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02beef6-cae3-4f87-a330-64b650e11d99-kube-api-access-757tw" (OuterVolumeSpecName: "kube-api-access-757tw") pod "c02beef6-cae3-4f87-a330-64b650e11d99" (UID: "c02beef6-cae3-4f87-a330-64b650e11d99"). InnerVolumeSpecName "kube-api-access-757tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:38:22.946694 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:22.946656    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-757tw\" (UniqueName: \"kubernetes.io/projected/c02beef6-cae3-4f87-a330-64b650e11d99-kube-api-access-757tw\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:38:22.946694 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:22.946689    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c02beef6-cae3-4f87-a330-64b650e11d99-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:38:22.946694 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:22.946700    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c02beef6-cae3-4f87-a330-64b650e11d99-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:38:22.946941 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:22.946709    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c02beef6-cae3-4f87-a330-64b650e11d99-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:38:22.946941 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:22.946719    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c02beef6-cae3-4f87-a330-64b650e11d99-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:38:23.011614 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.011581    2562 generic.go:356] "Generic (PLEG): container finished" podID="c02beef6-cae3-4f87-a330-64b650e11d99" containerID="de28458817852e82bc65b78f5bdc43a27f7dcc2ce8713d0de538187f26a52037" exitCode=0
Jul 21 17:38:23.011780 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.011640    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"
Jul 21 17:38:23.011780 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.011655    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" event={"ID":"c02beef6-cae3-4f87-a330-64b650e11d99","Type":"ContainerDied","Data":"de28458817852e82bc65b78f5bdc43a27f7dcc2ce8713d0de538187f26a52037"}
Jul 21 17:38:23.011780 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.011717    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg" event={"ID":"c02beef6-cae3-4f87-a330-64b650e11d99","Type":"ContainerDied","Data":"c576976de32d41a04682af9f655ba6baa2d3fa0293364047d43ead916d0964cf"}
Jul 21 17:38:23.011780 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.011748    2562 scope.go:117] "RemoveContainer" containerID="de28458817852e82bc65b78f5bdc43a27f7dcc2ce8713d0de538187f26a52037"
Jul 21 17:38:23.020810 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.020792    2562 scope.go:117] "RemoveContainer" containerID="543c5ebeeb9c4379981d4daeb1c698ba2cfe1e2cab6b4983d8d4a50862fa96ac"
Jul 21 17:38:23.028327 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.028300    2562 scope.go:117] "RemoveContainer" containerID="41c0524f3b79690bf6679f07362d4d06358fadaa056baf710e61d2408608bb2c"
Jul 21 17:38:23.033079 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.033044    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"]
Jul 21 17:38:23.035873 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.035850    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-dd6c55f7b-28bzg"]
Jul 21 17:38:23.037416 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.037389    2562 scope.go:117] "RemoveContainer" containerID="ca7e0b06b4f7bc5378498a87bcfac39474ce8c04c9315aaaa2d1f9032a6021f7"
Jul 21 17:38:23.045226 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.045206    2562 scope.go:117] "RemoveContainer" containerID="de28458817852e82bc65b78f5bdc43a27f7dcc2ce8713d0de538187f26a52037"
Jul 21 17:38:23.045537 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:38:23.045516    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de28458817852e82bc65b78f5bdc43a27f7dcc2ce8713d0de538187f26a52037\": container with ID starting with de28458817852e82bc65b78f5bdc43a27f7dcc2ce8713d0de538187f26a52037 not found: ID does not exist" containerID="de28458817852e82bc65b78f5bdc43a27f7dcc2ce8713d0de538187f26a52037"
Jul 21 17:38:23.045612 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.045548    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de28458817852e82bc65b78f5bdc43a27f7dcc2ce8713d0de538187f26a52037"} err="failed to get container status \"de28458817852e82bc65b78f5bdc43a27f7dcc2ce8713d0de538187f26a52037\": rpc error: code = NotFound desc = could not find container \"de28458817852e82bc65b78f5bdc43a27f7dcc2ce8713d0de538187f26a52037\": container with ID starting with de28458817852e82bc65b78f5bdc43a27f7dcc2ce8713d0de538187f26a52037 not found: ID does not exist"
Jul 21 17:38:23.045612 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.045569    2562 scope.go:117] "RemoveContainer" containerID="543c5ebeeb9c4379981d4daeb1c698ba2cfe1e2cab6b4983d8d4a50862fa96ac"
Jul 21 17:38:23.045845 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:38:23.045828    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543c5ebeeb9c4379981d4daeb1c698ba2cfe1e2cab6b4983d8d4a50862fa96ac\": container with ID starting with 543c5ebeeb9c4379981d4daeb1c698ba2cfe1e2cab6b4983d8d4a50862fa96ac not found: ID does not exist" containerID="543c5ebeeb9c4379981d4daeb1c698ba2cfe1e2cab6b4983d8d4a50862fa96ac"
Jul 21 17:38:23.045889 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.045851    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543c5ebeeb9c4379981d4daeb1c698ba2cfe1e2cab6b4983d8d4a50862fa96ac"} err="failed to get container status \"543c5ebeeb9c4379981d4daeb1c698ba2cfe1e2cab6b4983d8d4a50862fa96ac\": rpc error: code = NotFound desc = could not find container \"543c5ebeeb9c4379981d4daeb1c698ba2cfe1e2cab6b4983d8d4a50862fa96ac\": container with ID starting with 543c5ebeeb9c4379981d4daeb1c698ba2cfe1e2cab6b4983d8d4a50862fa96ac not found: ID does not exist"
Jul 21 17:38:23.045889 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.045866    2562 scope.go:117] "RemoveContainer" containerID="41c0524f3b79690bf6679f07362d4d06358fadaa056baf710e61d2408608bb2c"
Jul 21 17:38:23.046141 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:38:23.046123    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c0524f3b79690bf6679f07362d4d06358fadaa056baf710e61d2408608bb2c\": container with ID starting with 41c0524f3b79690bf6679f07362d4d06358fadaa056baf710e61d2408608bb2c not found: ID does not exist" containerID="41c0524f3b79690bf6679f07362d4d06358fadaa056baf710e61d2408608bb2c"
Jul 21 17:38:23.046219 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.046158    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c0524f3b79690bf6679f07362d4d06358fadaa056baf710e61d2408608bb2c"} err="failed to get container status \"41c0524f3b79690bf6679f07362d4d06358fadaa056baf710e61d2408608bb2c\": rpc error: code = NotFound desc = could not find container \"41c0524f3b79690bf6679f07362d4d06358fadaa056baf710e61d2408608bb2c\": container with ID starting with 41c0524f3b79690bf6679f07362d4d06358fadaa056baf710e61d2408608bb2c not found: ID does not exist"
Jul 21 17:38:23.046219 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.046179    2562 scope.go:117] "RemoveContainer" containerID="ca7e0b06b4f7bc5378498a87bcfac39474ce8c04c9315aaaa2d1f9032a6021f7"
Jul 21 17:38:23.046445 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:38:23.046428    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca7e0b06b4f7bc5378498a87bcfac39474ce8c04c9315aaaa2d1f9032a6021f7\": container with ID starting with ca7e0b06b4f7bc5378498a87bcfac39474ce8c04c9315aaaa2d1f9032a6021f7 not found: ID does not exist" containerID="ca7e0b06b4f7bc5378498a87bcfac39474ce8c04c9315aaaa2d1f9032a6021f7"
Jul 21 17:38:23.046491 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:23.046449    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7e0b06b4f7bc5378498a87bcfac39474ce8c04c9315aaaa2d1f9032a6021f7"} err="failed to get container status \"ca7e0b06b4f7bc5378498a87bcfac39474ce8c04c9315aaaa2d1f9032a6021f7\": rpc error: code = NotFound desc = could not find container \"ca7e0b06b4f7bc5378498a87bcfac39474ce8c04c9315aaaa2d1f9032a6021f7\": container with ID starting with ca7e0b06b4f7bc5378498a87bcfac39474ce8c04c9315aaaa2d1f9032a6021f7 not found: ID does not exist"
Jul 21 17:38:24.911731 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:24.911687    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Jul 21 17:38:24.912208 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:24.912190    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:38:24.921710 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:24.921685    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" path="/var/lib/kubelet/pods/c02beef6-cae3-4f87-a330-64b650e11d99/volumes"
Jul 21 17:38:34.911167 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:34.911125    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Jul 21 17:38:34.911652 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:34.911580    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:38:44.911172 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:44.911126    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Jul 21 17:38:44.911600 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:44.911482    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:38:54.911697 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:54.911646    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Jul 21 17:38:54.912198 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:38:54.912131    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:39:04.911811 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:04.911760    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Jul 21 17:39:04.912301 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:04.912168    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:39:09.916846 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:09.916771    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:39:09.917282 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:09.917122    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:39:17.671963 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:17.671930    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"]
Jul 21 17:39:17.672453 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:17.672198    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container" containerID="cri-o://82a3ef4660356731dcf09c2fce600703e4dcf3196d89106622879a35a066d6e3" gracePeriod=30
Jul 21 17:39:17.672453 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:17.672211    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="agent" containerID="cri-o://221d7d86171a86a7467c688f52159ea6bac14d8214498c6e68b61382d0c5aee4" gracePeriod=30
Jul 21 17:39:17.672453 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:17.672259    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kube-rbac-proxy" containerID="cri-o://0089cb8412e2a0833cfa2d527c82ba8282149d2c2b5f04e8461db2ccd0abd565" gracePeriod=30
Jul 21 17:39:18.249191 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:18.249153    2562 generic.go:356] "Generic (PLEG): container finished" podID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerID="0089cb8412e2a0833cfa2d527c82ba8282149d2c2b5f04e8461db2ccd0abd565" exitCode=2
Jul 21 17:39:18.249339 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:18.249226    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" event={"ID":"bbd232cd-437f-4e8e-9412-243a16cdc2c1","Type":"ContainerDied","Data":"0089cb8412e2a0833cfa2d527c82ba8282149d2c2b5f04e8461db2ccd0abd565"}
Jul 21 17:39:19.906870 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:19.906830    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 21 17:39:19.916292 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:19.916258    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Jul 21 17:39:19.916691 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:19.916668    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:39:22.267729 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:22.267696    2562 generic.go:356] "Generic (PLEG): container finished" podID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerID="82a3ef4660356731dcf09c2fce600703e4dcf3196d89106622879a35a066d6e3" exitCode=0
Jul 21 17:39:22.268120 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:22.267768    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" event={"ID":"bbd232cd-437f-4e8e-9412-243a16cdc2c1","Type":"ContainerDied","Data":"82a3ef4660356731dcf09c2fce600703e4dcf3196d89106622879a35a066d6e3"}
Jul 21 17:39:24.907529 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:24.907492    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 21 17:39:29.906841 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:29.906804    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 21 17:39:29.907241 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:29.906911    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:39:29.916673 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:29.916640    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Jul 21 17:39:29.917031 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:29.917012    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:39:34.907335 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:34.907295    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 21 17:39:37.868911 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.868883    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"]
Jul 21 17:39:37.869388 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.869234    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kube-rbac-proxy"
Jul 21 17:39:37.869388 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.869245    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kube-rbac-proxy"
Jul 21 17:39:37.869388 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.869252    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="agent"
Jul 21 17:39:37.869388 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.869257    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="agent"
Jul 21 17:39:37.869388 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.869266    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="storage-initializer"
Jul 21 17:39:37.869388 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.869273    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="storage-initializer"
Jul 21 17:39:37.869388 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.869298    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kserve-container"
Jul 21 17:39:37.869388 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.869303    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kserve-container"
Jul 21 17:39:37.869388 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.869357    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kube-rbac-proxy"
Jul 21 17:39:37.869388 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.869366    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="kserve-container"
Jul 21 17:39:37.869388 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.869373    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c02beef6-cae3-4f87-a330-64b650e11d99" containerName="agent"
Jul 21 17:39:37.872568 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.872552    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:37.875829 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.875809    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 21 17:39:37.876353 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.876318    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 21 17:39:37.886890 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.886869    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"]
Jul 21 17:39:37.963617 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.963591    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdfvq\" (UniqueName: \"kubernetes.io/projected/7b13850b-b34f-456c-96a2-b2762918beb3-kube-api-access-mdfvq\") pod \"isvc-logger-predictor-85db8844c-2srb5\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:37.963757 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.963624    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b13850b-b34f-456c-96a2-b2762918beb3-kserve-provision-location\") pod \"isvc-logger-predictor-85db8844c-2srb5\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:37.963757 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.963642    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b13850b-b34f-456c-96a2-b2762918beb3-cabundle-cert\") pod \"isvc-logger-predictor-85db8844c-2srb5\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:37.963757 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.963732    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b13850b-b34f-456c-96a2-b2762918beb3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-85db8844c-2srb5\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:37.963865 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:37.963779    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b13850b-b34f-456c-96a2-b2762918beb3-proxy-tls\") pod \"isvc-logger-predictor-85db8844c-2srb5\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:38.064904 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:38.064877    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b13850b-b34f-456c-96a2-b2762918beb3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-85db8844c-2srb5\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:38.065039 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:38.064921    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b13850b-b34f-456c-96a2-b2762918beb3-proxy-tls\") pod \"isvc-logger-predictor-85db8844c-2srb5\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:38.065039 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:38.064979    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdfvq\" (UniqueName: \"kubernetes.io/projected/7b13850b-b34f-456c-96a2-b2762918beb3-kube-api-access-mdfvq\") pod \"isvc-logger-predictor-85db8844c-2srb5\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:38.065039 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:38.065004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b13850b-b34f-456c-96a2-b2762918beb3-kserve-provision-location\") pod \"isvc-logger-predictor-85db8844c-2srb5\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:38.065199 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:38.065160    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b13850b-b34f-456c-96a2-b2762918beb3-cabundle-cert\") pod \"isvc-logger-predictor-85db8844c-2srb5\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:38.065533 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:38.065510    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b13850b-b34f-456c-96a2-b2762918beb3-kserve-provision-location\") pod \"isvc-logger-predictor-85db8844c-2srb5\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:38.065611 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:38.065601    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b13850b-b34f-456c-96a2-b2762918beb3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-85db8844c-2srb5\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:38.065808 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:38.065751    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b13850b-b34f-456c-96a2-b2762918beb3-cabundle-cert\") pod \"isvc-logger-predictor-85db8844c-2srb5\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:38.067534 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:38.067512    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b13850b-b34f-456c-96a2-b2762918beb3-proxy-tls\") pod \"isvc-logger-predictor-85db8844c-2srb5\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:38.075416 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:38.075398    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdfvq\" (UniqueName: \"kubernetes.io/projected/7b13850b-b34f-456c-96a2-b2762918beb3-kube-api-access-mdfvq\") pod \"isvc-logger-predictor-85db8844c-2srb5\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:38.182214 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:38.182189    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:38.324622 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:38.324594    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"]
Jul 21 17:39:38.326203 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:39:38.326175    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b13850b_b34f_456c_96a2_b2762918beb3.slice/crio-b812a80aa0deb7e5dff22652e6f973b3194b45b4ed6ab047bc7a957eb0e08034 WatchSource:0}: Error finding container b812a80aa0deb7e5dff22652e6f973b3194b45b4ed6ab047bc7a957eb0e08034: Status 404 returned error can't find the container with id b812a80aa0deb7e5dff22652e6f973b3194b45b4ed6ab047bc7a957eb0e08034
Jul 21 17:39:38.336316 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:38.336295    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" event={"ID":"7b13850b-b34f-456c-96a2-b2762918beb3","Type":"ContainerStarted","Data":"b812a80aa0deb7e5dff22652e6f973b3194b45b4ed6ab047bc7a957eb0e08034"}
Jul 21 17:39:39.342088 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:39.342034    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" event={"ID":"7b13850b-b34f-456c-96a2-b2762918beb3","Type":"ContainerStarted","Data":"c887cd4155ebfac0170aa8e4af00204d095c9859443a684388dc387e9c5e0532"}
Jul 21 17:39:39.906640 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:39.906603    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 21 17:39:39.917088 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:39.917032    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:5000: connect: connection refused"
Jul 21 17:39:39.917235 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:39.917173    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:39:39.917340 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:39.917315    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:39:39.917457 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:39.917443    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:39:42.356654 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:42.356625    2562 generic.go:356] "Generic (PLEG): container finished" podID="7b13850b-b34f-456c-96a2-b2762918beb3" containerID="c887cd4155ebfac0170aa8e4af00204d095c9859443a684388dc387e9c5e0532" exitCode=0
Jul 21 17:39:42.356654 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:42.356658    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" event={"ID":"7b13850b-b34f-456c-96a2-b2762918beb3","Type":"ContainerDied","Data":"c887cd4155ebfac0170aa8e4af00204d095c9859443a684388dc387e9c5e0532"}
Jul 21 17:39:43.363743 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:43.363712    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" event={"ID":"7b13850b-b34f-456c-96a2-b2762918beb3","Type":"ContainerStarted","Data":"4392136773a5f02efb5a3f0dc161e0db135e6d5ecf1af529f53ab7aa1964c0c8"}
Jul 21 17:39:43.364186 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:43.363774    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" event={"ID":"7b13850b-b34f-456c-96a2-b2762918beb3","Type":"ContainerStarted","Data":"920e1e27e73f6625e0d80fd3d053bf02b0fa35b2492e6694e8c90dac4e184396"}
Jul 21 17:39:43.364186 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:43.363799    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" event={"ID":"7b13850b-b34f-456c-96a2-b2762918beb3","Type":"ContainerStarted","Data":"83ffbd6876e619d0a20d5fda74f16c7c514e29399cc290906aa96eef8e572d07"}
Jul 21 17:39:43.364186 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:43.364109    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:43.364357 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:43.364193    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:43.364357 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:43.364230    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:43.365676 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:43.365652    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 21 17:39:43.366382 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:43.366357    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:39:43.387391 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:43.387347    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podStartSLOduration=2.911393928 podStartE2EDuration="6.387331351s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.475937423s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:39:43.385204495 +0000 UTC m=+834.956826270" watchObservedRunningTime="2026-07-21 17:39:43.387331351 +0000 UTC m=+834.958953106"
Jul 21 17:39:44.368256 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:44.368214    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 21 17:39:44.368704 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:44.368527    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:39:44.907273 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:44.907230    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 21 17:39:47.821110 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:47.821055    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:39:47.946781 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:47.946714    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjdkq\" (UniqueName: \"kubernetes.io/projected/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kube-api-access-tjdkq\") pod \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") "
Jul 21 17:39:47.946781 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:47.946762    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbd232cd-437f-4e8e-9412-243a16cdc2c1-proxy-tls\") pod \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") "
Jul 21 17:39:47.946986 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:47.946787    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kube-rbac-proxy-sar-config\") pod \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") "
Jul 21 17:39:47.946986 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:47.946818    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kserve-provision-location\") pod \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") "
Jul 21 17:39:47.946986 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:47.946868    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbd232cd-437f-4e8e-9412-243a16cdc2c1-cabundle-cert\") pod \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\" (UID: \"bbd232cd-437f-4e8e-9412-243a16cdc2c1\") "
Jul 21 17:39:47.947206 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:47.947179    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bbd232cd-437f-4e8e-9412-243a16cdc2c1" (UID: "bbd232cd-437f-4e8e-9412-243a16cdc2c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:39:47.947263 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:47.947211    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bbd232cd-437f-4e8e-9412-243a16cdc2c1" (UID: "bbd232cd-437f-4e8e-9412-243a16cdc2c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:39:47.947263 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:47.947246    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd232cd-437f-4e8e-9412-243a16cdc2c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bbd232cd-437f-4e8e-9412-243a16cdc2c1" (UID: "bbd232cd-437f-4e8e-9412-243a16cdc2c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:39:47.948965 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:47.948941    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kube-api-access-tjdkq" (OuterVolumeSpecName: "kube-api-access-tjdkq") pod "bbd232cd-437f-4e8e-9412-243a16cdc2c1" (UID: "bbd232cd-437f-4e8e-9412-243a16cdc2c1"). InnerVolumeSpecName "kube-api-access-tjdkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:39:47.949056 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:47.949031    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbd232cd-437f-4e8e-9412-243a16cdc2c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bbd232cd-437f-4e8e-9412-243a16cdc2c1" (UID: "bbd232cd-437f-4e8e-9412-243a16cdc2c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:39:48.048244 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.048215    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:39:48.048244 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.048240    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbd232cd-437f-4e8e-9412-243a16cdc2c1-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:39:48.048444 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.048254    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjdkq\" (UniqueName: \"kubernetes.io/projected/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kube-api-access-tjdkq\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:39:48.048444 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.048270    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbd232cd-437f-4e8e-9412-243a16cdc2c1-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:39:48.048444 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.048282    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbd232cd-437f-4e8e-9412-243a16cdc2c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:39:48.391693 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.391605    2562 generic.go:356] "Generic (PLEG): container finished" podID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerID="221d7d86171a86a7467c688f52159ea6bac14d8214498c6e68b61382d0c5aee4" exitCode=0
Jul 21 17:39:48.391693 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.391644    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" event={"ID":"bbd232cd-437f-4e8e-9412-243a16cdc2c1","Type":"ContainerDied","Data":"221d7d86171a86a7467c688f52159ea6bac14d8214498c6e68b61382d0c5aee4"}
Jul 21 17:39:48.391693 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.391691    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"
Jul 21 17:39:48.391891 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.391733    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5" event={"ID":"bbd232cd-437f-4e8e-9412-243a16cdc2c1","Type":"ContainerDied","Data":"70e4e479621beffde669681f25f7f4e7394cbf40311797ecc2b1afe398386878"}
Jul 21 17:39:48.391891 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.391784    2562 scope.go:117] "RemoveContainer" containerID="221d7d86171a86a7467c688f52159ea6bac14d8214498c6e68b61382d0c5aee4"
Jul 21 17:39:48.400680 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.400662    2562 scope.go:117] "RemoveContainer" containerID="0089cb8412e2a0833cfa2d527c82ba8282149d2c2b5f04e8461db2ccd0abd565"
Jul 21 17:39:48.407854 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.407840    2562 scope.go:117] "RemoveContainer" containerID="82a3ef4660356731dcf09c2fce600703e4dcf3196d89106622879a35a066d6e3"
Jul 21 17:39:48.413624 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.413603    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"]
Jul 21 17:39:48.415230 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.415215    2562 scope.go:117] "RemoveContainer" containerID="fcc1b901add4c4adac90d0a4aaaf7efc7ea38c0cba7d9c3bd616d36aaf34cbb3"
Jul 21 17:39:48.419154 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.419134    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7f965647f-8s5n5"]
Jul 21 17:39:48.422894 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.422878    2562 scope.go:117] "RemoveContainer" containerID="221d7d86171a86a7467c688f52159ea6bac14d8214498c6e68b61382d0c5aee4"
Jul 21 17:39:48.423155 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:39:48.423136    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"221d7d86171a86a7467c688f52159ea6bac14d8214498c6e68b61382d0c5aee4\": container with ID starting with 221d7d86171a86a7467c688f52159ea6bac14d8214498c6e68b61382d0c5aee4 not found: ID does not exist" containerID="221d7d86171a86a7467c688f52159ea6bac14d8214498c6e68b61382d0c5aee4"
Jul 21 17:39:48.423214 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.423163    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"221d7d86171a86a7467c688f52159ea6bac14d8214498c6e68b61382d0c5aee4"} err="failed to get container status \"221d7d86171a86a7467c688f52159ea6bac14d8214498c6e68b61382d0c5aee4\": rpc error: code = NotFound desc = could not find container \"221d7d86171a86a7467c688f52159ea6bac14d8214498c6e68b61382d0c5aee4\": container with ID starting with 221d7d86171a86a7467c688f52159ea6bac14d8214498c6e68b61382d0c5aee4 not found: ID does not exist"
Jul 21 17:39:48.423214 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.423180    2562 scope.go:117] "RemoveContainer" containerID="0089cb8412e2a0833cfa2d527c82ba8282149d2c2b5f04e8461db2ccd0abd565"
Jul 21 17:39:48.423357 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:39:48.423340    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0089cb8412e2a0833cfa2d527c82ba8282149d2c2b5f04e8461db2ccd0abd565\": container with ID starting with 0089cb8412e2a0833cfa2d527c82ba8282149d2c2b5f04e8461db2ccd0abd565 not found: ID does not exist" containerID="0089cb8412e2a0833cfa2d527c82ba8282149d2c2b5f04e8461db2ccd0abd565"
Jul 21 17:39:48.423397 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.423360    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0089cb8412e2a0833cfa2d527c82ba8282149d2c2b5f04e8461db2ccd0abd565"} err="failed to get container status \"0089cb8412e2a0833cfa2d527c82ba8282149d2c2b5f04e8461db2ccd0abd565\": rpc error: code = NotFound desc = could not find container \"0089cb8412e2a0833cfa2d527c82ba8282149d2c2b5f04e8461db2ccd0abd565\": container with ID starting with 0089cb8412e2a0833cfa2d527c82ba8282149d2c2b5f04e8461db2ccd0abd565 not found: ID does not exist"
Jul 21 17:39:48.423397 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.423372    2562 scope.go:117] "RemoveContainer" containerID="82a3ef4660356731dcf09c2fce600703e4dcf3196d89106622879a35a066d6e3"
Jul 21 17:39:48.423579 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:39:48.423561    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a3ef4660356731dcf09c2fce600703e4dcf3196d89106622879a35a066d6e3\": container with ID starting with 82a3ef4660356731dcf09c2fce600703e4dcf3196d89106622879a35a066d6e3 not found: ID does not exist" containerID="82a3ef4660356731dcf09c2fce600703e4dcf3196d89106622879a35a066d6e3"
Jul 21 17:39:48.423618 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.423576    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a3ef4660356731dcf09c2fce600703e4dcf3196d89106622879a35a066d6e3"} err="failed to get container status \"82a3ef4660356731dcf09c2fce600703e4dcf3196d89106622879a35a066d6e3\": rpc error: code = NotFound desc = could not find container \"82a3ef4660356731dcf09c2fce600703e4dcf3196d89106622879a35a066d6e3\": container with ID starting with 82a3ef4660356731dcf09c2fce600703e4dcf3196d89106622879a35a066d6e3 not found: ID does not exist"
Jul 21 17:39:48.423618 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.423587    2562 scope.go:117] "RemoveContainer" containerID="fcc1b901add4c4adac90d0a4aaaf7efc7ea38c0cba7d9c3bd616d36aaf34cbb3"
Jul 21 17:39:48.423822 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:39:48.423805    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc1b901add4c4adac90d0a4aaaf7efc7ea38c0cba7d9c3bd616d36aaf34cbb3\": container with ID starting with fcc1b901add4c4adac90d0a4aaaf7efc7ea38c0cba7d9c3bd616d36aaf34cbb3 not found: ID does not exist" containerID="fcc1b901add4c4adac90d0a4aaaf7efc7ea38c0cba7d9c3bd616d36aaf34cbb3"
Jul 21 17:39:48.423869 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.423837    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc1b901add4c4adac90d0a4aaaf7efc7ea38c0cba7d9c3bd616d36aaf34cbb3"} err="failed to get container status \"fcc1b901add4c4adac90d0a4aaaf7efc7ea38c0cba7d9c3bd616d36aaf34cbb3\": rpc error: code = NotFound desc = could not find container \"fcc1b901add4c4adac90d0a4aaaf7efc7ea38c0cba7d9c3bd616d36aaf34cbb3\": container with ID starting with fcc1b901add4c4adac90d0a4aaaf7efc7ea38c0cba7d9c3bd616d36aaf34cbb3 not found: ID does not exist"
Jul 21 17:39:48.920633 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:48.920607    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" path="/var/lib/kubelet/pods/bbd232cd-437f-4e8e-9412-243a16cdc2c1/volumes"
Jul 21 17:39:49.372121 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:49.372096    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:39:49.372671 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:49.372641    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 21 17:39:49.372936 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:49.372901    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:39:59.373439 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:59.373402    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 21 17:39:59.373894 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:39:59.373821    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:40:09.373267 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:40:09.373231    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 21 17:40:09.373797 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:40:09.373677    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:40:19.372707 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:40:19.372665    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 21 17:40:19.373124 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:40:19.373103    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:40:29.372993 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:40:29.372948    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 21 17:40:29.373489 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:40:29.373452    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:40:39.372715 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:40:39.372616    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 21 17:40:39.373200 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:40:39.373088    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:40:48.855486 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:40:48.855455    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 17:40:48.855965 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:40:48.855455    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 17:40:49.372644 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:40:49.372599    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 21 17:40:49.373142 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:40:49.373117    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:40:58.921390 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:40:58.921365    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:40:58.923673 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:40:58.921437    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:41:13.034548 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.034512    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"]
Jul 21 17:41:13.034957 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.034858    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kserve-container" containerID="cri-o://83ffbd6876e619d0a20d5fda74f16c7c514e29399cc290906aa96eef8e572d07" gracePeriod=30
Jul 21 17:41:13.034957 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.034891    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kube-rbac-proxy" containerID="cri-o://920e1e27e73f6625e0d80fd3d053bf02b0fa35b2492e6694e8c90dac4e184396" gracePeriod=30
Jul 21 17:41:13.035086 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.034886    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="agent" containerID="cri-o://4392136773a5f02efb5a3f0dc161e0db135e6d5ecf1af529f53ab7aa1964c0c8" gracePeriod=30
Jul 21 17:41:13.102406 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.102373    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"]
Jul 21 17:41:13.102706 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.102694    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container"
Jul 21 17:41:13.102759 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.102709    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container"
Jul 21 17:41:13.102759 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.102725    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kube-rbac-proxy"
Jul 21 17:41:13.102759 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.102730    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kube-rbac-proxy"
Jul 21 17:41:13.102759 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.102739    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="storage-initializer"
Jul 21 17:41:13.102759 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.102744    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="storage-initializer"
Jul 21 17:41:13.102759 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.102752    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="agent"
Jul 21 17:41:13.102759 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.102758    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="agent"
Jul 21 17:41:13.102990 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.102812    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="agent"
Jul 21 17:41:13.102990 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.102820    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kube-rbac-proxy"
Jul 21 17:41:13.102990 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.102826    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbd232cd-437f-4e8e-9412-243a16cdc2c1" containerName="kserve-container"
Jul 21 17:41:13.105834 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.105817    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.108289 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.108271    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 21 17:41:13.108405 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.108317    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 21 17:41:13.116249 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.116224    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"]
Jul 21 17:41:13.124110 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.124085    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ac9073c-5885-4e2e-9756-09f551071dbe-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bn4lr\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.124240 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.124157    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ac9073c-5885-4e2e-9756-09f551071dbe-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bn4lr\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.124240 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.124180    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvhg2\" (UniqueName: \"kubernetes.io/projected/9ac9073c-5885-4e2e-9756-09f551071dbe-kube-api-access-wvhg2\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bn4lr\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.124240 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.124199    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ac9073c-5885-4e2e-9756-09f551071dbe-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bn4lr\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.124240 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.124227    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ac9073c-5885-4e2e-9756-09f551071dbe-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bn4lr\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.225352 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.225313    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ac9073c-5885-4e2e-9756-09f551071dbe-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bn4lr\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.225352 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.225350    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvhg2\" (UniqueName: \"kubernetes.io/projected/9ac9073c-5885-4e2e-9756-09f551071dbe-kube-api-access-wvhg2\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bn4lr\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.225620 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.225369    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ac9073c-5885-4e2e-9756-09f551071dbe-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bn4lr\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.225620 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.225389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ac9073c-5885-4e2e-9756-09f551071dbe-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bn4lr\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.225620 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.225443    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ac9073c-5885-4e2e-9756-09f551071dbe-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bn4lr\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.225847 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.225823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ac9073c-5885-4e2e-9756-09f551071dbe-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bn4lr\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.226025 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.226006    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ac9073c-5885-4e2e-9756-09f551071dbe-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bn4lr\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.226286 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.226271    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ac9073c-5885-4e2e-9756-09f551071dbe-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bn4lr\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.228146 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.228126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ac9073c-5885-4e2e-9756-09f551071dbe-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bn4lr\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.234612 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.234587    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvhg2\" (UniqueName: \"kubernetes.io/projected/9ac9073c-5885-4e2e-9756-09f551071dbe-kube-api-access-wvhg2\") pod \"isvc-autogluon-v1-predictor-574b5b7495-bn4lr\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.416809 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.416763    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:41:13.557873 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.557841    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"]
Jul 21 17:41:13.560942 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:41:13.560919    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac9073c_5885_4e2e_9756_09f551071dbe.slice/crio-1cc5d2419fbe7185cd9872fe55372a61c4c10804bc3e606987abd01087014917 WatchSource:0}: Error finding container 1cc5d2419fbe7185cd9872fe55372a61c4c10804bc3e606987abd01087014917: Status 404 returned error can't find the container with id 1cc5d2419fbe7185cd9872fe55372a61c4c10804bc3e606987abd01087014917
Jul 21 17:41:13.751231 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.751146    2562 generic.go:356] "Generic (PLEG): container finished" podID="7b13850b-b34f-456c-96a2-b2762918beb3" containerID="920e1e27e73f6625e0d80fd3d053bf02b0fa35b2492e6694e8c90dac4e184396" exitCode=2
Jul 21 17:41:13.751231 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.751219    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" event={"ID":"7b13850b-b34f-456c-96a2-b2762918beb3","Type":"ContainerDied","Data":"920e1e27e73f6625e0d80fd3d053bf02b0fa35b2492e6694e8c90dac4e184396"}
Jul 21 17:41:13.752772 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.752748    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr" event={"ID":"9ac9073c-5885-4e2e-9756-09f551071dbe","Type":"ContainerStarted","Data":"1af5ff8471df0e8afc751ab5c95959310d870ef8984aca08dba90ca927914733"}
Jul 21 17:41:13.752914 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:13.752808    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr" event={"ID":"9ac9073c-5885-4e2e-9756-09f551071dbe","Type":"ContainerStarted","Data":"1cc5d2419fbe7185cd9872fe55372a61c4c10804bc3e606987abd01087014917"}
Jul 21 17:41:14.368380 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:14.368324    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 21 17:41:17.773247 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:17.773210    2562 generic.go:356] "Generic (PLEG): container finished" podID="7b13850b-b34f-456c-96a2-b2762918beb3" containerID="83ffbd6876e619d0a20d5fda74f16c7c514e29399cc290906aa96eef8e572d07" exitCode=0
Jul 21 17:41:17.773247 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:17.773248    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" event={"ID":"7b13850b-b34f-456c-96a2-b2762918beb3","Type":"ContainerDied","Data":"83ffbd6876e619d0a20d5fda74f16c7c514e29399cc290906aa96eef8e572d07"}
Jul 21 17:41:18.918590 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:18.918546    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 21 17:41:18.919033 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:18.918883    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:41:19.369223 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:19.369136    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 21 17:41:24.368944 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:24.368898    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 21 17:41:24.369410 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:24.369035    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:41:28.918413 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:28.918375    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 21 17:41:28.918911 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:28.918824    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:41:29.368719 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:29.368622    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 21 17:41:34.368769 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:34.368666    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 21 17:41:38.918686 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:38.918640    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 21 17:41:38.919147 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:38.919035    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 21 17:41:38.921488 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:38.921470    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:41:38.921571 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:38.921548    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:41:39.368845 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:39.368749    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 21 17:41:42.883787 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:42.883753    2562 generic.go:356] "Generic (PLEG): container finished" podID="9ac9073c-5885-4e2e-9756-09f551071dbe" containerID="1af5ff8471df0e8afc751ab5c95959310d870ef8984aca08dba90ca927914733" exitCode=0
Jul 21 17:41:42.884243 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:42.883829    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr" event={"ID":"9ac9073c-5885-4e2e-9756-09f551071dbe","Type":"ContainerDied","Data":"1af5ff8471df0e8afc751ab5c95959310d870ef8984aca08dba90ca927914733"}
Jul 21 17:41:42.885237 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:42.885222    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 17:41:43.195230 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.195201    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:41:43.280051 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.280004    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b13850b-b34f-456c-96a2-b2762918beb3-cabundle-cert\") pod \"7b13850b-b34f-456c-96a2-b2762918beb3\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") "
Jul 21 17:41:43.280244 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.280083    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b13850b-b34f-456c-96a2-b2762918beb3-proxy-tls\") pod \"7b13850b-b34f-456c-96a2-b2762918beb3\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") "
Jul 21 17:41:43.280244 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.280121    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdfvq\" (UniqueName: \"kubernetes.io/projected/7b13850b-b34f-456c-96a2-b2762918beb3-kube-api-access-mdfvq\") pod \"7b13850b-b34f-456c-96a2-b2762918beb3\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") "
Jul 21 17:41:43.280244 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.280150    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b13850b-b34f-456c-96a2-b2762918beb3-kube-rbac-proxy-sar-config\") pod \"7b13850b-b34f-456c-96a2-b2762918beb3\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") "
Jul 21 17:41:43.280244 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.280210    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b13850b-b34f-456c-96a2-b2762918beb3-kserve-provision-location\") pod \"7b13850b-b34f-456c-96a2-b2762918beb3\" (UID: \"7b13850b-b34f-456c-96a2-b2762918beb3\") "
Jul 21 17:41:43.280473 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.280367    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b13850b-b34f-456c-96a2-b2762918beb3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7b13850b-b34f-456c-96a2-b2762918beb3" (UID: "7b13850b-b34f-456c-96a2-b2762918beb3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:41:43.280603 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.280579    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b13850b-b34f-456c-96a2-b2762918beb3-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:41:43.280679 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.280625    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b13850b-b34f-456c-96a2-b2762918beb3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b13850b-b34f-456c-96a2-b2762918beb3" (UID: "7b13850b-b34f-456c-96a2-b2762918beb3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:41:43.280679 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.280635    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b13850b-b34f-456c-96a2-b2762918beb3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b13850b-b34f-456c-96a2-b2762918beb3" (UID: "7b13850b-b34f-456c-96a2-b2762918beb3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:41:43.283148 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.283112    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b13850b-b34f-456c-96a2-b2762918beb3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b13850b-b34f-456c-96a2-b2762918beb3" (UID: "7b13850b-b34f-456c-96a2-b2762918beb3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:41:43.283401 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.283379    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b13850b-b34f-456c-96a2-b2762918beb3-kube-api-access-mdfvq" (OuterVolumeSpecName: "kube-api-access-mdfvq") pod "7b13850b-b34f-456c-96a2-b2762918beb3" (UID: "7b13850b-b34f-456c-96a2-b2762918beb3"). InnerVolumeSpecName "kube-api-access-mdfvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:41:43.381317 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.381266    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdfvq\" (UniqueName: \"kubernetes.io/projected/7b13850b-b34f-456c-96a2-b2762918beb3-kube-api-access-mdfvq\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:41:43.381317 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.381305    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b13850b-b34f-456c-96a2-b2762918beb3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:41:43.381317 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.381320    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b13850b-b34f-456c-96a2-b2762918beb3-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:41:43.381574 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.381337    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b13850b-b34f-456c-96a2-b2762918beb3-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:41:43.893751 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.893713    2562 generic.go:356] "Generic (PLEG): container finished" podID="7b13850b-b34f-456c-96a2-b2762918beb3" containerID="4392136773a5f02efb5a3f0dc161e0db135e6d5ecf1af529f53ab7aa1964c0c8" exitCode=0
Jul 21 17:41:43.894256 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.893767    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"
Jul 21 17:41:43.894256 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.893867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" event={"ID":"7b13850b-b34f-456c-96a2-b2762918beb3","Type":"ContainerDied","Data":"4392136773a5f02efb5a3f0dc161e0db135e6d5ecf1af529f53ab7aa1964c0c8"}
Jul 21 17:41:43.894256 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.893960    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5" event={"ID":"7b13850b-b34f-456c-96a2-b2762918beb3","Type":"ContainerDied","Data":"b812a80aa0deb7e5dff22652e6f973b3194b45b4ed6ab047bc7a957eb0e08034"}
Jul 21 17:41:43.894256 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.894006    2562 scope.go:117] "RemoveContainer" containerID="4392136773a5f02efb5a3f0dc161e0db135e6d5ecf1af529f53ab7aa1964c0c8"
Jul 21 17:41:43.921461 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.921427    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"]
Jul 21 17:41:43.925231 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:43.925180    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85db8844c-2srb5"]
Jul 21 17:41:44.654043 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:44.654019    2562 scope.go:117] "RemoveContainer" containerID="920e1e27e73f6625e0d80fd3d053bf02b0fa35b2492e6694e8c90dac4e184396"
Jul 21 17:41:44.921826 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:44.921737    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" path="/var/lib/kubelet/pods/7b13850b-b34f-456c-96a2-b2762918beb3/volumes"
Jul 21 17:41:45.046961 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:45.046822    2562 scope.go:117] "RemoveContainer" containerID="83ffbd6876e619d0a20d5fda74f16c7c514e29399cc290906aa96eef8e572d07"
Jul 21 17:41:45.147534 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:45.147508    2562 scope.go:117] "RemoveContainer" containerID="c887cd4155ebfac0170aa8e4af00204d095c9859443a684388dc387e9c5e0532"
Jul 21 17:41:45.242084 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:45.242051    2562 scope.go:117] "RemoveContainer" containerID="4392136773a5f02efb5a3f0dc161e0db135e6d5ecf1af529f53ab7aa1964c0c8"
Jul 21 17:41:45.242415 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:41:45.242394    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4392136773a5f02efb5a3f0dc161e0db135e6d5ecf1af529f53ab7aa1964c0c8\": container with ID starting with 4392136773a5f02efb5a3f0dc161e0db135e6d5ecf1af529f53ab7aa1964c0c8 not found: ID does not exist" containerID="4392136773a5f02efb5a3f0dc161e0db135e6d5ecf1af529f53ab7aa1964c0c8"
Jul 21 17:41:45.242471 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:45.242426    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4392136773a5f02efb5a3f0dc161e0db135e6d5ecf1af529f53ab7aa1964c0c8"} err="failed to get container status \"4392136773a5f02efb5a3f0dc161e0db135e6d5ecf1af529f53ab7aa1964c0c8\": rpc error: code = NotFound desc = could not find container \"4392136773a5f02efb5a3f0dc161e0db135e6d5ecf1af529f53ab7aa1964c0c8\": container with ID starting with 4392136773a5f02efb5a3f0dc161e0db135e6d5ecf1af529f53ab7aa1964c0c8 not found: ID does not exist"
Jul 21 17:41:45.242471 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:45.242446    2562 scope.go:117] "RemoveContainer" containerID="920e1e27e73f6625e0d80fd3d053bf02b0fa35b2492e6694e8c90dac4e184396"
Jul 21 17:41:45.242710 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:41:45.242693    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"920e1e27e73f6625e0d80fd3d053bf02b0fa35b2492e6694e8c90dac4e184396\": container with ID starting with 920e1e27e73f6625e0d80fd3d053bf02b0fa35b2492e6694e8c90dac4e184396 not found: ID does not exist" containerID="920e1e27e73f6625e0d80fd3d053bf02b0fa35b2492e6694e8c90dac4e184396"
Jul 21 17:41:45.242745 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:45.242715    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"920e1e27e73f6625e0d80fd3d053bf02b0fa35b2492e6694e8c90dac4e184396"} err="failed to get container status \"920e1e27e73f6625e0d80fd3d053bf02b0fa35b2492e6694e8c90dac4e184396\": rpc error: code = NotFound desc = could not find container \"920e1e27e73f6625e0d80fd3d053bf02b0fa35b2492e6694e8c90dac4e184396\": container with ID starting with 920e1e27e73f6625e0d80fd3d053bf02b0fa35b2492e6694e8c90dac4e184396 not found: ID does not exist"
Jul 21 17:41:45.242745 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:45.242728    2562 scope.go:117] "RemoveContainer" containerID="83ffbd6876e619d0a20d5fda74f16c7c514e29399cc290906aa96eef8e572d07"
Jul 21 17:41:45.242961 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:41:45.242947    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ffbd6876e619d0a20d5fda74f16c7c514e29399cc290906aa96eef8e572d07\": container with ID starting with 83ffbd6876e619d0a20d5fda74f16c7c514e29399cc290906aa96eef8e572d07 not found: ID does not exist" containerID="83ffbd6876e619d0a20d5fda74f16c7c514e29399cc290906aa96eef8e572d07"
Jul 21 17:41:45.243002 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:45.242964    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ffbd6876e619d0a20d5fda74f16c7c514e29399cc290906aa96eef8e572d07"} err="failed to get container status \"83ffbd6876e619d0a20d5fda74f16c7c514e29399cc290906aa96eef8e572d07\": rpc error: code = NotFound desc = could not find container \"83ffbd6876e619d0a20d5fda74f16c7c514e29399cc290906aa96eef8e572d07\": container with ID starting with 83ffbd6876e619d0a20d5fda74f16c7c514e29399cc290906aa96eef8e572d07 not found: ID does not exist"
Jul 21 17:41:45.243002 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:45.242974    2562 scope.go:117] "RemoveContainer" containerID="c887cd4155ebfac0170aa8e4af00204d095c9859443a684388dc387e9c5e0532"
Jul 21 17:41:45.243197 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:41:45.243178    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c887cd4155ebfac0170aa8e4af00204d095c9859443a684388dc387e9c5e0532\": container with ID starting with c887cd4155ebfac0170aa8e4af00204d095c9859443a684388dc387e9c5e0532 not found: ID does not exist" containerID="c887cd4155ebfac0170aa8e4af00204d095c9859443a684388dc387e9c5e0532"
Jul 21 17:41:45.243239 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:41:45.243205    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c887cd4155ebfac0170aa8e4af00204d095c9859443a684388dc387e9c5e0532"} err="failed to get container status \"c887cd4155ebfac0170aa8e4af00204d095c9859443a684388dc387e9c5e0532\": rpc error: code = NotFound desc = could not find container \"c887cd4155ebfac0170aa8e4af00204d095c9859443a684388dc387e9c5e0532\": container with ID starting with c887cd4155ebfac0170aa8e4af00204d095c9859443a684388dc387e9c5e0532 not found: ID does not exist"
Jul 21 17:43:51.494811 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:43:51.494772    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr" event={"ID":"9ac9073c-5885-4e2e-9756-09f551071dbe","Type":"ContainerStarted","Data":"15bb5f3d892d1e8228bc2fd44e2d5fe0f459956514ea651f6f35ecc348775da9"}
Jul 21 17:43:51.495358 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:43:51.494846    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr" event={"ID":"9ac9073c-5885-4e2e-9756-09f551071dbe","Type":"ContainerStarted","Data":"d0d3b7e0cd9064c7826cc0144bf7385bd9fa5a7feb660e6cb69c7057fd242896"}
Jul 21 17:43:51.495358 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:43:51.495012    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:43:51.524245 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:43:51.524189    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr" podStartSLOduration=1.687758622 podStartE2EDuration="2m38.524173553s" totalImagesPullingTime="2m7.71061864s" totalInitContainerRuntime="29.125796291s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:43:51.522253683 +0000 UTC m=+1083.093875479" watchObservedRunningTime="2026-07-21 17:43:51.524173553 +0000 UTC m=+1083.095795307"
Jul 21 17:43:52.498875 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:43:52.498836    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:43:52.500145 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:43:52.500118    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr" podUID="9ac9073c-5885-4e2e-9756-09f551071dbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 21 17:43:53.502543 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:43:53.502500    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr" podUID="9ac9073c-5885-4e2e-9756-09f551071dbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 21 17:43:58.508620 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:43:58.508589    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:43:58.509244 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:43:58.509216    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr" podUID="9ac9073c-5885-4e2e-9756-09f551071dbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 21 17:44:08.509890 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:08.509860    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:44:19.597258 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:19.597218    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"]
Jul 21 17:44:19.597621 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:19.597489    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr" podUID="9ac9073c-5885-4e2e-9756-09f551071dbe" containerName="kserve-container" containerID="cri-o://d0d3b7e0cd9064c7826cc0144bf7385bd9fa5a7feb660e6cb69c7057fd242896" gracePeriod=30
Jul 21 17:44:19.597621 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:19.597505    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr" podUID="9ac9073c-5885-4e2e-9756-09f551071dbe" containerName="kube-rbac-proxy" containerID="cri-o://15bb5f3d892d1e8228bc2fd44e2d5fe0f459956514ea651f6f35ecc348775da9" gracePeriod=30
Jul 21 17:44:20.627293 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:20.627260    2562 generic.go:356] "Generic (PLEG): container finished" podID="9ac9073c-5885-4e2e-9756-09f551071dbe" containerID="15bb5f3d892d1e8228bc2fd44e2d5fe0f459956514ea651f6f35ecc348775da9" exitCode=2
Jul 21 17:44:20.627722 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:20.627339    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr" event={"ID":"9ac9073c-5885-4e2e-9756-09f551071dbe","Type":"ContainerDied","Data":"15bb5f3d892d1e8228bc2fd44e2d5fe0f459956514ea651f6f35ecc348775da9"}
Jul 21 17:44:21.174544 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.174521    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:44:21.239786 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.239706    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvhg2\" (UniqueName: \"kubernetes.io/projected/9ac9073c-5885-4e2e-9756-09f551071dbe-kube-api-access-wvhg2\") pod \"9ac9073c-5885-4e2e-9756-09f551071dbe\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") "
Jul 21 17:44:21.239786 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.239748    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ac9073c-5885-4e2e-9756-09f551071dbe-proxy-tls\") pod \"9ac9073c-5885-4e2e-9756-09f551071dbe\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") "
Jul 21 17:44:21.239786 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.239781    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ac9073c-5885-4e2e-9756-09f551071dbe-kserve-provision-location\") pod \"9ac9073c-5885-4e2e-9756-09f551071dbe\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") "
Jul 21 17:44:21.240050 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.239806    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ac9073c-5885-4e2e-9756-09f551071dbe-kube-rbac-proxy-sar-config\") pod \"9ac9073c-5885-4e2e-9756-09f551071dbe\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") "
Jul 21 17:44:21.240050 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.239846    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ac9073c-5885-4e2e-9756-09f551071dbe-cabundle-cert\") pod \"9ac9073c-5885-4e2e-9756-09f551071dbe\" (UID: \"9ac9073c-5885-4e2e-9756-09f551071dbe\") "
Jul 21 17:44:21.240251 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.240221    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac9073c-5885-4e2e-9756-09f551071dbe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9ac9073c-5885-4e2e-9756-09f551071dbe" (UID: "9ac9073c-5885-4e2e-9756-09f551071dbe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:44:21.240298 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.240226    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac9073c-5885-4e2e-9756-09f551071dbe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9ac9073c-5885-4e2e-9756-09f551071dbe" (UID: "9ac9073c-5885-4e2e-9756-09f551071dbe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:44:21.242082 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.242026    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac9073c-5885-4e2e-9756-09f551071dbe-kube-api-access-wvhg2" (OuterVolumeSpecName: "kube-api-access-wvhg2") pod "9ac9073c-5885-4e2e-9756-09f551071dbe" (UID: "9ac9073c-5885-4e2e-9756-09f551071dbe"). InnerVolumeSpecName "kube-api-access-wvhg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:44:21.242082 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.242034    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac9073c-5885-4e2e-9756-09f551071dbe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ac9073c-5885-4e2e-9756-09f551071dbe" (UID: "9ac9073c-5885-4e2e-9756-09f551071dbe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:44:21.244311 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.244289    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac9073c-5885-4e2e-9756-09f551071dbe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9ac9073c-5885-4e2e-9756-09f551071dbe" (UID: "9ac9073c-5885-4e2e-9756-09f551071dbe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:44:21.340439 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.340402    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvhg2\" (UniqueName: \"kubernetes.io/projected/9ac9073c-5885-4e2e-9756-09f551071dbe-kube-api-access-wvhg2\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:44:21.340439 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.340431    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ac9073c-5885-4e2e-9756-09f551071dbe-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:44:21.340439 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.340441    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ac9073c-5885-4e2e-9756-09f551071dbe-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:44:21.340439 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.340449    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ac9073c-5885-4e2e-9756-09f551071dbe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:44:21.340699 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.340458    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ac9073c-5885-4e2e-9756-09f551071dbe-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:44:21.633611 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.633529    2562 generic.go:356] "Generic (PLEG): container finished" podID="9ac9073c-5885-4e2e-9756-09f551071dbe" containerID="d0d3b7e0cd9064c7826cc0144bf7385bd9fa5a7feb660e6cb69c7057fd242896" exitCode=0
Jul 21 17:44:21.633611 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.633583    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"
Jul 21 17:44:21.634209 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.633615    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr" event={"ID":"9ac9073c-5885-4e2e-9756-09f551071dbe","Type":"ContainerDied","Data":"d0d3b7e0cd9064c7826cc0144bf7385bd9fa5a7feb660e6cb69c7057fd242896"}
Jul 21 17:44:21.634209 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.633686    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr" event={"ID":"9ac9073c-5885-4e2e-9756-09f551071dbe","Type":"ContainerDied","Data":"1cc5d2419fbe7185cd9872fe55372a61c4c10804bc3e606987abd01087014917"}
Jul 21 17:44:21.634209 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.633726    2562 scope.go:117] "RemoveContainer" containerID="15bb5f3d892d1e8228bc2fd44e2d5fe0f459956514ea651f6f35ecc348775da9"
Jul 21 17:44:21.642685 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.642671    2562 scope.go:117] "RemoveContainer" containerID="d0d3b7e0cd9064c7826cc0144bf7385bd9fa5a7feb660e6cb69c7057fd242896"
Jul 21 17:44:21.650710 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.650695    2562 scope.go:117] "RemoveContainer" containerID="1af5ff8471df0e8afc751ab5c95959310d870ef8984aca08dba90ca927914733"
Jul 21 17:44:21.654855 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.654829    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"]
Jul 21 17:44:21.658471 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.658453    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-bn4lr"]
Jul 21 17:44:21.659147 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.659130    2562 scope.go:117] "RemoveContainer" containerID="15bb5f3d892d1e8228bc2fd44e2d5fe0f459956514ea651f6f35ecc348775da9"
Jul 21 17:44:21.659419 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:44:21.659400    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15bb5f3d892d1e8228bc2fd44e2d5fe0f459956514ea651f6f35ecc348775da9\": container with ID starting with 15bb5f3d892d1e8228bc2fd44e2d5fe0f459956514ea651f6f35ecc348775da9 not found: ID does not exist" containerID="15bb5f3d892d1e8228bc2fd44e2d5fe0f459956514ea651f6f35ecc348775da9"
Jul 21 17:44:21.659479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.659428    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bb5f3d892d1e8228bc2fd44e2d5fe0f459956514ea651f6f35ecc348775da9"} err="failed to get container status \"15bb5f3d892d1e8228bc2fd44e2d5fe0f459956514ea651f6f35ecc348775da9\": rpc error: code = NotFound desc = could not find container \"15bb5f3d892d1e8228bc2fd44e2d5fe0f459956514ea651f6f35ecc348775da9\": container with ID starting with 15bb5f3d892d1e8228bc2fd44e2d5fe0f459956514ea651f6f35ecc348775da9 not found: ID does not exist"
Jul 21 17:44:21.659479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.659446    2562 scope.go:117] "RemoveContainer" containerID="d0d3b7e0cd9064c7826cc0144bf7385bd9fa5a7feb660e6cb69c7057fd242896"
Jul 21 17:44:21.659700 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:44:21.659683    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d3b7e0cd9064c7826cc0144bf7385bd9fa5a7feb660e6cb69c7057fd242896\": container with ID starting with d0d3b7e0cd9064c7826cc0144bf7385bd9fa5a7feb660e6cb69c7057fd242896 not found: ID does not exist" containerID="d0d3b7e0cd9064c7826cc0144bf7385bd9fa5a7feb660e6cb69c7057fd242896"
Jul 21 17:44:21.659740 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.659707    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d3b7e0cd9064c7826cc0144bf7385bd9fa5a7feb660e6cb69c7057fd242896"} err="failed to get container status \"d0d3b7e0cd9064c7826cc0144bf7385bd9fa5a7feb660e6cb69c7057fd242896\": rpc error: code = NotFound desc = could not find container \"d0d3b7e0cd9064c7826cc0144bf7385bd9fa5a7feb660e6cb69c7057fd242896\": container with ID starting with d0d3b7e0cd9064c7826cc0144bf7385bd9fa5a7feb660e6cb69c7057fd242896 not found: ID does not exist"
Jul 21 17:44:21.659740 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.659729    2562 scope.go:117] "RemoveContainer" containerID="1af5ff8471df0e8afc751ab5c95959310d870ef8984aca08dba90ca927914733"
Jul 21 17:44:21.659954 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:44:21.659937    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af5ff8471df0e8afc751ab5c95959310d870ef8984aca08dba90ca927914733\": container with ID starting with 1af5ff8471df0e8afc751ab5c95959310d870ef8984aca08dba90ca927914733 not found: ID does not exist" containerID="1af5ff8471df0e8afc751ab5c95959310d870ef8984aca08dba90ca927914733"
Jul 21 17:44:21.659993 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:21.659958    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af5ff8471df0e8afc751ab5c95959310d870ef8984aca08dba90ca927914733"} err="failed to get container status \"1af5ff8471df0e8afc751ab5c95959310d870ef8984aca08dba90ca927914733\": rpc error: code = NotFound desc = could not find container \"1af5ff8471df0e8afc751ab5c95959310d870ef8984aca08dba90ca927914733\": container with ID starting with 1af5ff8471df0e8afc751ab5c95959310d870ef8984aca08dba90ca927914733 not found: ID does not exist"
Jul 21 17:44:22.921145 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:22.921113    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ac9073c-5885-4e2e-9756-09f551071dbe" path="/var/lib/kubelet/pods/9ac9073c-5885-4e2e-9756-09f551071dbe/volumes"
Jul 21 17:44:24.644743 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.644710    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"]
Jul 21 17:44:24.645187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645027    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ac9073c-5885-4e2e-9756-09f551071dbe" containerName="kserve-container"
Jul 21 17:44:24.645187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645039    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ac9073c-5885-4e2e-9756-09f551071dbe" containerName="kserve-container"
Jul 21 17:44:24.645187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645051    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ac9073c-5885-4e2e-9756-09f551071dbe" containerName="storage-initializer"
Jul 21 17:44:24.645187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645057    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ac9073c-5885-4e2e-9756-09f551071dbe" containerName="storage-initializer"
Jul 21 17:44:24.645187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645083    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="agent"
Jul 21 17:44:24.645187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645092    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="agent"
Jul 21 17:44:24.645187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645106    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kserve-container"
Jul 21 17:44:24.645187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645111    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kserve-container"
Jul 21 17:44:24.645187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645123    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ac9073c-5885-4e2e-9756-09f551071dbe" containerName="kube-rbac-proxy"
Jul 21 17:44:24.645187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645128    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ac9073c-5885-4e2e-9756-09f551071dbe" containerName="kube-rbac-proxy"
Jul 21 17:44:24.645187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645133    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kube-rbac-proxy"
Jul 21 17:44:24.645187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645138    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kube-rbac-proxy"
Jul 21 17:44:24.645187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645148    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="storage-initializer"
Jul 21 17:44:24.645187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645153    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="storage-initializer"
Jul 21 17:44:24.645628 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645218    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="agent"
Jul 21 17:44:24.645628 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645232    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kserve-container"
Jul 21 17:44:24.645628 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645239    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ac9073c-5885-4e2e-9756-09f551071dbe" containerName="kube-rbac-proxy"
Jul 21 17:44:24.645628 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645246    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b13850b-b34f-456c-96a2-b2762918beb3" containerName="kube-rbac-proxy"
Jul 21 17:44:24.645628 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.645256    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ac9073c-5885-4e2e-9756-09f551071dbe" containerName="kserve-container"
Jul 21 17:44:24.669048 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.669015    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"]
Jul 21 17:44:24.669199 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.669163    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:24.671992 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.671969    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 17:44:24.672151 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.671969    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 17:44:24.672151 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.671969    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 17:44:24.672151 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.672144    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 17:44:24.672321 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.672145    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 17:44:24.673146 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.673131    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 21 17:44:24.673247 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.673201    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 21 17:44:24.767754 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.767719    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-x2cj4\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:24.767754 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.767750    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g72tq\" (UniqueName: \"kubernetes.io/projected/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kube-api-access-g72tq\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-x2cj4\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:24.768027 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.767775    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-x2cj4\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:24.768027 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.767819    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-x2cj4\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:24.768027 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.767865    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-x2cj4\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:24.868437 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.868407    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-x2cj4\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:24.868437 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.868438    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g72tq\" (UniqueName: \"kubernetes.io/projected/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kube-api-access-g72tq\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-x2cj4\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:24.868623 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.868458    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-x2cj4\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:24.868623 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.868574    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-x2cj4\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:24.868717 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.868650    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-x2cj4\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:24.869010 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.868987    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-x2cj4\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:24.869316 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.869295    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-x2cj4\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:24.869366 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.869295    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-x2cj4\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:24.870985 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.870969    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-x2cj4\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:24.877637 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.877607    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g72tq\" (UniqueName: \"kubernetes.io/projected/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kube-api-access-g72tq\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-x2cj4\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:24.979602 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:24.979564    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:25.131547 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:25.131521    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"]
Jul 21 17:44:25.134598 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:44:25.134573    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2d3e05_154a_4316_9a31_3ac6d2f5acba.slice/crio-6871dc7f04340439374dcfbbe1be21f8c83d5e8e51298a2e132e089b354da95f WatchSource:0}: Error finding container 6871dc7f04340439374dcfbbe1be21f8c83d5e8e51298a2e132e089b354da95f: Status 404 returned error can't find the container with id 6871dc7f04340439374dcfbbe1be21f8c83d5e8e51298a2e132e089b354da95f
Jul 21 17:44:25.658110 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:25.658079    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4" event={"ID":"ea2d3e05-154a-4316-9a31-3ac6d2f5acba","Type":"ContainerStarted","Data":"6a08b773f8c4abac35ee081b65cd7df6e8254aa28e0cff19b198e07815983bac"}
Jul 21 17:44:25.658479 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:25.658137    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4" event={"ID":"ea2d3e05-154a-4316-9a31-3ac6d2f5acba","Type":"ContainerStarted","Data":"6871dc7f04340439374dcfbbe1be21f8c83d5e8e51298a2e132e089b354da95f"}
Jul 21 17:44:54.779275 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:54.779242    2562 generic.go:356] "Generic (PLEG): container finished" podID="ea2d3e05-154a-4316-9a31-3ac6d2f5acba" containerID="6a08b773f8c4abac35ee081b65cd7df6e8254aa28e0cff19b198e07815983bac" exitCode=0
Jul 21 17:44:54.779702 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:54.779322    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4" event={"ID":"ea2d3e05-154a-4316-9a31-3ac6d2f5acba","Type":"ContainerDied","Data":"6a08b773f8c4abac35ee081b65cd7df6e8254aa28e0cff19b198e07815983bac"}
Jul 21 17:44:55.786781 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:55.786748    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4" event={"ID":"ea2d3e05-154a-4316-9a31-3ac6d2f5acba","Type":"ContainerStarted","Data":"91925666045c8b19ff07a10b81d40d3b02d62e4620aeff6a60872f8a8a9ed316"}
Jul 21 17:44:55.787253 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:55.786809    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4" event={"ID":"ea2d3e05-154a-4316-9a31-3ac6d2f5acba","Type":"ContainerStarted","Data":"f33396eb75d119bbbc58d0f2ca90bac401e70a8de5f296dcf30159154283c9f8"}
Jul 21 17:44:55.787253 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:55.787008    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:55.787253 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:55.787049    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:44:55.807632 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:44:55.807588    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4" podStartSLOduration=3.083411189 podStartE2EDuration="31.807572133s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.724160944s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:44:55.805599249 +0000 UTC m=+1147.377221021" watchObservedRunningTime="2026-07-21 17:44:55.807572133 +0000 UTC m=+1147.379193888"
Jul 21 17:45:01.797875 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:45:01.797841    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:45:48.885158 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:45:48.885129    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 17:45:48.888447 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:45:48.888420    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 17:46:26.925740 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:26.925711    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:46:37.958284 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:37.958229    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4" podUID="ea2d3e05-154a-4316-9a31-3ac6d2f5acba" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.31:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 21 17:46:40.990754 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:40.990719    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"]
Jul 21 17:46:40.991231 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:40.991020    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4" podUID="ea2d3e05-154a-4316-9a31-3ac6d2f5acba" containerName="kserve-container" containerID="cri-o://f33396eb75d119bbbc58d0f2ca90bac401e70a8de5f296dcf30159154283c9f8" gracePeriod=30
Jul 21 17:46:40.991297 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:40.991214    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4" podUID="ea2d3e05-154a-4316-9a31-3ac6d2f5acba" containerName="kube-rbac-proxy" containerID="cri-o://91925666045c8b19ff07a10b81d40d3b02d62e4620aeff6a60872f8a8a9ed316" gracePeriod=30
Jul 21 17:46:41.212884 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:41.212849    2562 generic.go:356] "Generic (PLEG): container finished" podID="ea2d3e05-154a-4316-9a31-3ac6d2f5acba" containerID="91925666045c8b19ff07a10b81d40d3b02d62e4620aeff6a60872f8a8a9ed316" exitCode=2
Jul 21 17:46:41.213096 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:41.212912    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4" event={"ID":"ea2d3e05-154a-4316-9a31-3ac6d2f5acba","Type":"ContainerDied","Data":"91925666045c8b19ff07a10b81d40d3b02d62e4620aeff6a60872f8a8a9ed316"}
Jul 21 17:46:41.792181 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:41.792131    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4" podUID="ea2d3e05-154a-4316-9a31-3ac6d2f5acba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 21 17:46:42.190791 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:46:42.190761    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2d3e05_154a_4316_9a31_3ac6d2f5acba.slice/crio-f33396eb75d119bbbc58d0f2ca90bac401e70a8de5f296dcf30159154283c9f8.scope\": RecentStats: unable to find data in memory cache]"
Jul 21 17:46:42.220367 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.220332    2562 generic.go:356] "Generic (PLEG): container finished" podID="ea2d3e05-154a-4316-9a31-3ac6d2f5acba" containerID="f33396eb75d119bbbc58d0f2ca90bac401e70a8de5f296dcf30159154283c9f8" exitCode=0
Jul 21 17:46:42.220509 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.220405    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4" event={"ID":"ea2d3e05-154a-4316-9a31-3ac6d2f5acba","Type":"ContainerDied","Data":"f33396eb75d119bbbc58d0f2ca90bac401e70a8de5f296dcf30159154283c9f8"}
Jul 21 17:46:42.333720 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.333698    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:46:42.471084 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.470979    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g72tq\" (UniqueName: \"kubernetes.io/projected/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kube-api-access-g72tq\") pod \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") "
Jul 21 17:46:42.471084 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.471022    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kserve-provision-location\") pod \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") "
Jul 21 17:46:42.471084 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.471039    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kube-rbac-proxy-sar-config\") pod \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") "
Jul 21 17:46:42.471334 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.471125    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-proxy-tls\") pod \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") "
Jul 21 17:46:42.471334 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.471151    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-cabundle-cert\") pod \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\" (UID: \"ea2d3e05-154a-4316-9a31-3ac6d2f5acba\") "
Jul 21 17:46:42.471522 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.471493    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ea2d3e05-154a-4316-9a31-3ac6d2f5acba" (UID: "ea2d3e05-154a-4316-9a31-3ac6d2f5acba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:46:42.471630 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.471578    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ea2d3e05-154a-4316-9a31-3ac6d2f5acba" (UID: "ea2d3e05-154a-4316-9a31-3ac6d2f5acba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:46:42.473448 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.473427    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea2d3e05-154a-4316-9a31-3ac6d2f5acba" (UID: "ea2d3e05-154a-4316-9a31-3ac6d2f5acba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:46:42.473448 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.473431    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kube-api-access-g72tq" (OuterVolumeSpecName: "kube-api-access-g72tq") pod "ea2d3e05-154a-4316-9a31-3ac6d2f5acba" (UID: "ea2d3e05-154a-4316-9a31-3ac6d2f5acba"). InnerVolumeSpecName "kube-api-access-g72tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:46:42.475410 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.475384    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea2d3e05-154a-4316-9a31-3ac6d2f5acba" (UID: "ea2d3e05-154a-4316-9a31-3ac6d2f5acba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:46:42.571877 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.571844    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:46:42.571877 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.571869    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:46:42.571877 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.571878    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g72tq\" (UniqueName: \"kubernetes.io/projected/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kube-api-access-g72tq\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:46:42.571877 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.571887    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:46:42.572153 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:42.571896    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea2d3e05-154a-4316-9a31-3ac6d2f5acba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:46:43.225931 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:43.225846    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"
Jul 21 17:46:43.225931 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:43.225870    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4" event={"ID":"ea2d3e05-154a-4316-9a31-3ac6d2f5acba","Type":"ContainerDied","Data":"6871dc7f04340439374dcfbbe1be21f8c83d5e8e51298a2e132e089b354da95f"}
Jul 21 17:46:43.225931 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:43.225930    2562 scope.go:117] "RemoveContainer" containerID="91925666045c8b19ff07a10b81d40d3b02d62e4620aeff6a60872f8a8a9ed316"
Jul 21 17:46:43.234374 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:43.234348    2562 scope.go:117] "RemoveContainer" containerID="f33396eb75d119bbbc58d0f2ca90bac401e70a8de5f296dcf30159154283c9f8"
Jul 21 17:46:43.241815 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:43.241796    2562 scope.go:117] "RemoveContainer" containerID="6a08b773f8c4abac35ee081b65cd7df6e8254aa28e0cff19b198e07815983bac"
Jul 21 17:46:43.244024 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:43.244002    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"]
Jul 21 17:46:43.250594 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:43.250570    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-x2cj4"]
Jul 21 17:46:44.920785 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:46:44.920753    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea2d3e05-154a-4316-9a31-3ac6d2f5acba" path="/var/lib/kubelet/pods/ea2d3e05-154a-4316-9a31-3ac6d2f5acba/volumes"
Jul 21 17:50:48.914933 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:50:48.914906    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 17:50:48.919568 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:50:48.919539    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 17:51:08.342324 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.342292    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"]
Jul 21 17:51:08.342750 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.342666    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2d3e05-154a-4316-9a31-3ac6d2f5acba" containerName="kube-rbac-proxy"
Jul 21 17:51:08.342750 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.342684    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2d3e05-154a-4316-9a31-3ac6d2f5acba" containerName="kube-rbac-proxy"
Jul 21 17:51:08.342750 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.342704    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2d3e05-154a-4316-9a31-3ac6d2f5acba" containerName="kserve-container"
Jul 21 17:51:08.342750 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.342710    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2d3e05-154a-4316-9a31-3ac6d2f5acba" containerName="kserve-container"
Jul 21 17:51:08.342750 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.342722    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2d3e05-154a-4316-9a31-3ac6d2f5acba" containerName="storage-initializer"
Jul 21 17:51:08.342750 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.342729    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2d3e05-154a-4316-9a31-3ac6d2f5acba" containerName="storage-initializer"
Jul 21 17:51:08.342931 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.342790    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2d3e05-154a-4316-9a31-3ac6d2f5acba" containerName="kserve-container"
Jul 21 17:51:08.342931 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.342798    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2d3e05-154a-4316-9a31-3ac6d2f5acba" containerName="kube-rbac-proxy"
Jul 21 17:51:08.344673 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.344658    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.347473 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.347445    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 17:51:08.347613 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.347477    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 17:51:08.347613 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.347494    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 17:51:08.347751 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.347735    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 21 17:51:08.347977 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.347959    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 21 17:51:08.348660 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.348638    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 17:51:08.348660 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.348643    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 17:51:08.356787 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.356766    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"]
Jul 21 17:51:08.442659 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.442622    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1097b635-0e8c-47b5-a400-d7e045b2b49c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.442659 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.442657    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1097b635-0e8c-47b5-a400-d7e045b2b49c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.442878 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.442685    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmnfr\" (UniqueName: \"kubernetes.io/projected/1097b635-0e8c-47b5-a400-d7e045b2b49c-kube-api-access-bmnfr\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.442878 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.442784    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1097b635-0e8c-47b5-a400-d7e045b2b49c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.442878 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.442816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1097b635-0e8c-47b5-a400-d7e045b2b49c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.543459 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.543419    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1097b635-0e8c-47b5-a400-d7e045b2b49c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.543459 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.543464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1097b635-0e8c-47b5-a400-d7e045b2b49c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.543707 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.543507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1097b635-0e8c-47b5-a400-d7e045b2b49c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.543707 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.543525    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1097b635-0e8c-47b5-a400-d7e045b2b49c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.543707 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.543546    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmnfr\" (UniqueName: \"kubernetes.io/projected/1097b635-0e8c-47b5-a400-d7e045b2b49c-kube-api-access-bmnfr\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.543957 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.543937    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1097b635-0e8c-47b5-a400-d7e045b2b49c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.544291 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.544267    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1097b635-0e8c-47b5-a400-d7e045b2b49c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.544349 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.544294    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1097b635-0e8c-47b5-a400-d7e045b2b49c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.546321 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.546263    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1097b635-0e8c-47b5-a400-d7e045b2b49c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.551920 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.551897    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmnfr\" (UniqueName: \"kubernetes.io/projected/1097b635-0e8c-47b5-a400-d7e045b2b49c-kube-api-access-bmnfr\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.655156 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.655118    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:08.790772 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:08.790740    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"]
Jul 21 17:51:08.792973 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:51:08.792939    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1097b635_0e8c_47b5_a400_d7e045b2b49c.slice/crio-d9a4b52e82941586e83133e3800045dc0f00de5438a20c65f9b84f88fec3282f WatchSource:0}: Error finding container d9a4b52e82941586e83133e3800045dc0f00de5438a20c65f9b84f88fec3282f: Status 404 returned error can't find the container with id d9a4b52e82941586e83133e3800045dc0f00de5438a20c65f9b84f88fec3282f
Jul 21 17:51:09.270690 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:09.270548    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g" event={"ID":"1097b635-0e8c-47b5-a400-d7e045b2b49c","Type":"ContainerStarted","Data":"16d2dd41578711c8f96574f254e608e5e94283d92b99b1f88cc5bc591c79457a"}
Jul 21 17:51:09.270690 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:09.270627    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g" event={"ID":"1097b635-0e8c-47b5-a400-d7e045b2b49c","Type":"ContainerStarted","Data":"d9a4b52e82941586e83133e3800045dc0f00de5438a20c65f9b84f88fec3282f"}
Jul 21 17:51:38.389106 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:38.389050    2562 generic.go:356] "Generic (PLEG): container finished" podID="1097b635-0e8c-47b5-a400-d7e045b2b49c" containerID="16d2dd41578711c8f96574f254e608e5e94283d92b99b1f88cc5bc591c79457a" exitCode=0
Jul 21 17:51:38.389497 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:38.389127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g" event={"ID":"1097b635-0e8c-47b5-a400-d7e045b2b49c","Type":"ContainerDied","Data":"16d2dd41578711c8f96574f254e608e5e94283d92b99b1f88cc5bc591c79457a"}
Jul 21 17:51:39.394738 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:39.394706    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g" event={"ID":"1097b635-0e8c-47b5-a400-d7e045b2b49c","Type":"ContainerStarted","Data":"3f5514ee9d977127aa22da36b2893aace3854a732d9c5af949408885bef53b1e"}
Jul 21 17:51:39.395170 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:39.394767    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g" event={"ID":"1097b635-0e8c-47b5-a400-d7e045b2b49c","Type":"ContainerStarted","Data":"807d8f8ed44eb4d4141dd49994d83d24fa14a3d492a05011a8b8c7251860bf6f"}
Jul 21 17:51:39.395170 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:39.394948    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:39.423284 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:39.423238    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g" podStartSLOduration=2.7740492850000003 podStartE2EDuration="31.423225727s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.649176442s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:51:39.421985884 +0000 UTC m=+1550.993607639" watchObservedRunningTime="2026-07-21 17:51:39.423225727 +0000 UTC m=+1550.994847481"
Jul 21 17:51:40.398802 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:40.398764    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:51:46.409508 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:51:46.409483    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:53:13.920266 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:13.920219    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:53:24.704157 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:24.704121    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"]
Jul 21 17:53:24.704696 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:24.704501    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g" podUID="1097b635-0e8c-47b5-a400-d7e045b2b49c" containerName="kserve-container" containerID="cri-o://807d8f8ed44eb4d4141dd49994d83d24fa14a3d492a05011a8b8c7251860bf6f" gracePeriod=30
Jul 21 17:53:24.704696 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:24.704535    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g" podUID="1097b635-0e8c-47b5-a400-d7e045b2b49c" containerName="kube-rbac-proxy" containerID="cri-o://3f5514ee9d977127aa22da36b2893aace3854a732d9c5af949408885bef53b1e" gracePeriod=30
Jul 21 17:53:25.819380 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:25.819346    2562 generic.go:356] "Generic (PLEG): container finished" podID="1097b635-0e8c-47b5-a400-d7e045b2b49c" containerID="3f5514ee9d977127aa22da36b2893aace3854a732d9c5af949408885bef53b1e" exitCode=2
Jul 21 17:53:25.819785 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:25.819427    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g" event={"ID":"1097b635-0e8c-47b5-a400-d7e045b2b49c","Type":"ContainerDied","Data":"3f5514ee9d977127aa22da36b2893aace3854a732d9c5af949408885bef53b1e"}
Jul 21 17:53:26.146470 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.146447    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:53:26.248104 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.248043    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1097b635-0e8c-47b5-a400-d7e045b2b49c-kserve-provision-location\") pod \"1097b635-0e8c-47b5-a400-d7e045b2b49c\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") "
Jul 21 17:53:26.248104 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.248124    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1097b635-0e8c-47b5-a400-d7e045b2b49c-kube-rbac-proxy-sar-config\") pod \"1097b635-0e8c-47b5-a400-d7e045b2b49c\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") "
Jul 21 17:53:26.248370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.248151    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1097b635-0e8c-47b5-a400-d7e045b2b49c-proxy-tls\") pod \"1097b635-0e8c-47b5-a400-d7e045b2b49c\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") "
Jul 21 17:53:26.248370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.248173    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmnfr\" (UniqueName: \"kubernetes.io/projected/1097b635-0e8c-47b5-a400-d7e045b2b49c-kube-api-access-bmnfr\") pod \"1097b635-0e8c-47b5-a400-d7e045b2b49c\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") "
Jul 21 17:53:26.248370 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.248204    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1097b635-0e8c-47b5-a400-d7e045b2b49c-cabundle-cert\") pod \"1097b635-0e8c-47b5-a400-d7e045b2b49c\" (UID: \"1097b635-0e8c-47b5-a400-d7e045b2b49c\") "
Jul 21 17:53:26.248605 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.248575    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1097b635-0e8c-47b5-a400-d7e045b2b49c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1097b635-0e8c-47b5-a400-d7e045b2b49c" (UID: "1097b635-0e8c-47b5-a400-d7e045b2b49c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:53:26.248687 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.248608    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1097b635-0e8c-47b5-a400-d7e045b2b49c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1097b635-0e8c-47b5-a400-d7e045b2b49c" (UID: "1097b635-0e8c-47b5-a400-d7e045b2b49c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:53:26.250406 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.250374    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1097b635-0e8c-47b5-a400-d7e045b2b49c-kube-api-access-bmnfr" (OuterVolumeSpecName: "kube-api-access-bmnfr") pod "1097b635-0e8c-47b5-a400-d7e045b2b49c" (UID: "1097b635-0e8c-47b5-a400-d7e045b2b49c"). InnerVolumeSpecName "kube-api-access-bmnfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:53:26.250513 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.250416    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1097b635-0e8c-47b5-a400-d7e045b2b49c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1097b635-0e8c-47b5-a400-d7e045b2b49c" (UID: "1097b635-0e8c-47b5-a400-d7e045b2b49c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:53:26.252744 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.252724    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1097b635-0e8c-47b5-a400-d7e045b2b49c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1097b635-0e8c-47b5-a400-d7e045b2b49c" (UID: "1097b635-0e8c-47b5-a400-d7e045b2b49c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:53:26.349253 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.349173    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1097b635-0e8c-47b5-a400-d7e045b2b49c-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:53:26.349253 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.349200    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1097b635-0e8c-47b5-a400-d7e045b2b49c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:53:26.349253 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.349209    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1097b635-0e8c-47b5-a400-d7e045b2b49c-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:53:26.349253 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.349219    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmnfr\" (UniqueName: \"kubernetes.io/projected/1097b635-0e8c-47b5-a400-d7e045b2b49c-kube-api-access-bmnfr\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:53:26.349253 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.349228    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1097b635-0e8c-47b5-a400-d7e045b2b49c-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:53:26.831411 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.831375    2562 generic.go:356] "Generic (PLEG): container finished" podID="1097b635-0e8c-47b5-a400-d7e045b2b49c" containerID="807d8f8ed44eb4d4141dd49994d83d24fa14a3d492a05011a8b8c7251860bf6f" exitCode=0
Jul 21 17:53:26.831893 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.831456    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g" event={"ID":"1097b635-0e8c-47b5-a400-d7e045b2b49c","Type":"ContainerDied","Data":"807d8f8ed44eb4d4141dd49994d83d24fa14a3d492a05011a8b8c7251860bf6f"}
Jul 21 17:53:26.831893 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.831482    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"
Jul 21 17:53:26.831893 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.831536    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g" event={"ID":"1097b635-0e8c-47b5-a400-d7e045b2b49c","Type":"ContainerDied","Data":"d9a4b52e82941586e83133e3800045dc0f00de5438a20c65f9b84f88fec3282f"}
Jul 21 17:53:26.831893 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.831567    2562 scope.go:117] "RemoveContainer" containerID="3f5514ee9d977127aa22da36b2893aace3854a732d9c5af949408885bef53b1e"
Jul 21 17:53:26.840486 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.840457    2562 scope.go:117] "RemoveContainer" containerID="807d8f8ed44eb4d4141dd49994d83d24fa14a3d492a05011a8b8c7251860bf6f"
Jul 21 17:53:26.848233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.848214    2562 scope.go:117] "RemoveContainer" containerID="16d2dd41578711c8f96574f254e608e5e94283d92b99b1f88cc5bc591c79457a"
Jul 21 17:53:26.854286 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.854260    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"]
Jul 21 17:53:26.857202 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.857181    2562 scope.go:117] "RemoveContainer" containerID="3f5514ee9d977127aa22da36b2893aace3854a732d9c5af949408885bef53b1e"
Jul 21 17:53:26.857523 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:53:26.857501    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f5514ee9d977127aa22da36b2893aace3854a732d9c5af949408885bef53b1e\": container with ID starting with 3f5514ee9d977127aa22da36b2893aace3854a732d9c5af949408885bef53b1e not found: ID does not exist" containerID="3f5514ee9d977127aa22da36b2893aace3854a732d9c5af949408885bef53b1e"
Jul 21 17:53:26.857613 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.857531    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f5514ee9d977127aa22da36b2893aace3854a732d9c5af949408885bef53b1e"} err="failed to get container status \"3f5514ee9d977127aa22da36b2893aace3854a732d9c5af949408885bef53b1e\": rpc error: code = NotFound desc = could not find container \"3f5514ee9d977127aa22da36b2893aace3854a732d9c5af949408885bef53b1e\": container with ID starting with 3f5514ee9d977127aa22da36b2893aace3854a732d9c5af949408885bef53b1e not found: ID does not exist"
Jul 21 17:53:26.857613 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.857551    2562 scope.go:117] "RemoveContainer" containerID="807d8f8ed44eb4d4141dd49994d83d24fa14a3d492a05011a8b8c7251860bf6f"
Jul 21 17:53:26.857880 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:53:26.857857    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"807d8f8ed44eb4d4141dd49994d83d24fa14a3d492a05011a8b8c7251860bf6f\": container with ID starting with 807d8f8ed44eb4d4141dd49994d83d24fa14a3d492a05011a8b8c7251860bf6f not found: ID does not exist" containerID="807d8f8ed44eb4d4141dd49994d83d24fa14a3d492a05011a8b8c7251860bf6f"
Jul 21 17:53:26.857920 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.857890    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"807d8f8ed44eb4d4141dd49994d83d24fa14a3d492a05011a8b8c7251860bf6f"} err="failed to get container status \"807d8f8ed44eb4d4141dd49994d83d24fa14a3d492a05011a8b8c7251860bf6f\": rpc error: code = NotFound desc = could not find container \"807d8f8ed44eb4d4141dd49994d83d24fa14a3d492a05011a8b8c7251860bf6f\": container with ID starting with 807d8f8ed44eb4d4141dd49994d83d24fa14a3d492a05011a8b8c7251860bf6f not found: ID does not exist"
Jul 21 17:53:26.857920 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.857912    2562 scope.go:117] "RemoveContainer" containerID="16d2dd41578711c8f96574f254e608e5e94283d92b99b1f88cc5bc591c79457a"
Jul 21 17:53:26.857984 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.857938    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g798g"]
Jul 21 17:53:26.858214 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:53:26.858195    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d2dd41578711c8f96574f254e608e5e94283d92b99b1f88cc5bc591c79457a\": container with ID starting with 16d2dd41578711c8f96574f254e608e5e94283d92b99b1f88cc5bc591c79457a not found: ID does not exist" containerID="16d2dd41578711c8f96574f254e608e5e94283d92b99b1f88cc5bc591c79457a"
Jul 21 17:53:26.858274 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.858220    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d2dd41578711c8f96574f254e608e5e94283d92b99b1f88cc5bc591c79457a"} err="failed to get container status \"16d2dd41578711c8f96574f254e608e5e94283d92b99b1f88cc5bc591c79457a\": rpc error: code = NotFound desc = could not find container \"16d2dd41578711c8f96574f254e608e5e94283d92b99b1f88cc5bc591c79457a\": container with ID starting with 16d2dd41578711c8f96574f254e608e5e94283d92b99b1f88cc5bc591c79457a not found: ID does not exist"
Jul 21 17:53:26.921802 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:26.921770    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1097b635-0e8c-47b5-a400-d7e045b2b49c" path="/var/lib/kubelet/pods/1097b635-0e8c-47b5-a400-d7e045b2b49c/volumes"
Jul 21 17:53:29.972734 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.972699    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"]
Jul 21 17:53:29.973244 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.973024    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1097b635-0e8c-47b5-a400-d7e045b2b49c" containerName="storage-initializer"
Jul 21 17:53:29.973244 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.973035    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1097b635-0e8c-47b5-a400-d7e045b2b49c" containerName="storage-initializer"
Jul 21 17:53:29.973244 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.973052    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1097b635-0e8c-47b5-a400-d7e045b2b49c" containerName="kserve-container"
Jul 21 17:53:29.973244 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.973057    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1097b635-0e8c-47b5-a400-d7e045b2b49c" containerName="kserve-container"
Jul 21 17:53:29.973244 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.973081    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1097b635-0e8c-47b5-a400-d7e045b2b49c" containerName="kube-rbac-proxy"
Jul 21 17:53:29.973244 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.973087    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1097b635-0e8c-47b5-a400-d7e045b2b49c" containerName="kube-rbac-proxy"
Jul 21 17:53:29.973244 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.973140    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1097b635-0e8c-47b5-a400-d7e045b2b49c" containerName="kube-rbac-proxy"
Jul 21 17:53:29.973244 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.973150    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1097b635-0e8c-47b5-a400-d7e045b2b49c" containerName="kserve-container"
Jul 21 17:53:29.977834 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.977814    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:29.980755 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.980724    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 21 17:53:29.980888 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.980822    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 17:53:29.980943 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.980913    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 17:53:29.981181 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.981164    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 21 17:53:29.981919 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.981899    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 17:53:29.981919 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.981909    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 17:53:29.982034 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.981941    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 17:53:29.988334 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:29.988300    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"]
Jul 21 17:53:30.081742 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.081707    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c9bm\" (UniqueName: \"kubernetes.io/projected/d0ba2aea-3426-4deb-937b-fe0954cb0372-kube-api-access-8c9bm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:30.081914 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.081751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0ba2aea-3426-4deb-937b-fe0954cb0372-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:30.081914 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.081818    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0ba2aea-3426-4deb-937b-fe0954cb0372-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:30.081914 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.081847    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0ba2aea-3426-4deb-937b-fe0954cb0372-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:30.082019 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.081924    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0ba2aea-3426-4deb-937b-fe0954cb0372-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:30.182595 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.182557    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0ba2aea-3426-4deb-937b-fe0954cb0372-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:30.182595 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.182596    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0ba2aea-3426-4deb-937b-fe0954cb0372-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:30.182843 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.182731    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0ba2aea-3426-4deb-937b-fe0954cb0372-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:30.182843 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.182808    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9bm\" (UniqueName: \"kubernetes.io/projected/d0ba2aea-3426-4deb-937b-fe0954cb0372-kube-api-access-8c9bm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:30.182942 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.182846    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0ba2aea-3426-4deb-937b-fe0954cb0372-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:30.183194 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.183157    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0ba2aea-3426-4deb-937b-fe0954cb0372-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:30.183414 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.183384    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0ba2aea-3426-4deb-937b-fe0954cb0372-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:30.183522 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.183412    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0ba2aea-3426-4deb-937b-fe0954cb0372-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:30.185416 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.185388    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0ba2aea-3426-4deb-937b-fe0954cb0372-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:30.192632 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.192613    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c9bm\" (UniqueName: \"kubernetes.io/projected/d0ba2aea-3426-4deb-937b-fe0954cb0372-kube-api-access-8c9bm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:30.289476 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.289385    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:30.427892 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.427868    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"]
Jul 21 17:53:30.429860 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:53:30.429827    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ba2aea_3426_4deb_937b_fe0954cb0372.slice/crio-1a9d8e12ad47c633d2f054447e27ac9125d7747a2e6dc3584aaa1ff9dea09136 WatchSource:0}: Error finding container 1a9d8e12ad47c633d2f054447e27ac9125d7747a2e6dc3584aaa1ff9dea09136: Status 404 returned error can't find the container with id 1a9d8e12ad47c633d2f054447e27ac9125d7747a2e6dc3584aaa1ff9dea09136
Jul 21 17:53:30.853596 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.853561    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g" event={"ID":"d0ba2aea-3426-4deb-937b-fe0954cb0372","Type":"ContainerStarted","Data":"04ca3909846f1449ed48e710477b06ad51bb175458c3579d71c45d063bb38430"}
Jul 21 17:53:30.853750 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:30.853626    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g" event={"ID":"d0ba2aea-3426-4deb-937b-fe0954cb0372","Type":"ContainerStarted","Data":"1a9d8e12ad47c633d2f054447e27ac9125d7747a2e6dc3584aaa1ff9dea09136"}
Jul 21 17:53:39.892383 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:39.892340    2562 generic.go:356] "Generic (PLEG): container finished" podID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerID="04ca3909846f1449ed48e710477b06ad51bb175458c3579d71c45d063bb38430" exitCode=0
Jul 21 17:53:39.892789 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:39.892397    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g" event={"ID":"d0ba2aea-3426-4deb-937b-fe0954cb0372","Type":"ContainerDied","Data":"04ca3909846f1449ed48e710477b06ad51bb175458c3579d71c45d063bb38430"}
Jul 21 17:53:40.899277 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:40.899240    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g" event={"ID":"d0ba2aea-3426-4deb-937b-fe0954cb0372","Type":"ContainerStarted","Data":"99dc80fe36ef97122c0032e477e75f618e5cef27b25cfb565a29ed10e9718e04"}
Jul 21 17:53:40.899671 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:40.899305    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g" event={"ID":"d0ba2aea-3426-4deb-937b-fe0954cb0372","Type":"ContainerStarted","Data":"77c0708404cd6688d75234661fe13903cb5fecd6c865c0aa0231ce26ec430880"}
Jul 21 17:53:40.899671 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:40.899509    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:40.921233 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:40.921183    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g" podStartSLOduration=3.059304231 podStartE2EDuration="11.921165864s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.861861633s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:53:40.918904844 +0000 UTC m=+1672.490526601" watchObservedRunningTime="2026-07-21 17:53:40.921165864 +0000 UTC m=+1672.492787619"
Jul 21 17:53:41.905090 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:41.905040    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:41.906698 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:41.906667    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g" podUID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 21 17:53:42.908865 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:42.908822    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g" podUID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 21 17:53:47.914130 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:47.914098    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:53:47.914768 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:47.914737    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g" podUID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 21 17:53:57.916230 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:53:57.916202    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:54:12.357386 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:12.357296    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"]
Jul 21 17:54:12.357839 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:12.357621    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g" podUID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerName="kserve-container" containerID="cri-o://77c0708404cd6688d75234661fe13903cb5fecd6c865c0aa0231ce26ec430880" gracePeriod=30
Jul 21 17:54:12.357839 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:12.357624    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g" podUID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerName="kube-rbac-proxy" containerID="cri-o://99dc80fe36ef97122c0032e477e75f618e5cef27b25cfb565a29ed10e9718e04" gracePeriod=30
Jul 21 17:54:12.909022 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:12.908983    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g" podUID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 21 17:54:13.037755 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:13.037719    2562 generic.go:356] "Generic (PLEG): container finished" podID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerID="99dc80fe36ef97122c0032e477e75f618e5cef27b25cfb565a29ed10e9718e04" exitCode=2
Jul 21 17:54:13.037933 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:13.037795    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g" event={"ID":"d0ba2aea-3426-4deb-937b-fe0954cb0372","Type":"ContainerDied","Data":"99dc80fe36ef97122c0032e477e75f618e5cef27b25cfb565a29ed10e9718e04"}
Jul 21 17:54:14.046022 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.045993    2562 generic.go:356] "Generic (PLEG): container finished" podID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerID="77c0708404cd6688d75234661fe13903cb5fecd6c865c0aa0231ce26ec430880" exitCode=0
Jul 21 17:54:14.046398 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.046103    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g" event={"ID":"d0ba2aea-3426-4deb-937b-fe0954cb0372","Type":"ContainerDied","Data":"77c0708404cd6688d75234661fe13903cb5fecd6c865c0aa0231ce26ec430880"}
Jul 21 17:54:14.105058 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.105034    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:54:14.172163 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.172130    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0ba2aea-3426-4deb-937b-fe0954cb0372-kube-rbac-proxy-sar-config\") pod \"d0ba2aea-3426-4deb-937b-fe0954cb0372\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") "
Jul 21 17:54:14.172163 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.172167    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0ba2aea-3426-4deb-937b-fe0954cb0372-kserve-provision-location\") pod \"d0ba2aea-3426-4deb-937b-fe0954cb0372\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") "
Jul 21 17:54:14.172417 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.172212    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0ba2aea-3426-4deb-937b-fe0954cb0372-proxy-tls\") pod \"d0ba2aea-3426-4deb-937b-fe0954cb0372\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") "
Jul 21 17:54:14.172417 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.172234    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c9bm\" (UniqueName: \"kubernetes.io/projected/d0ba2aea-3426-4deb-937b-fe0954cb0372-kube-api-access-8c9bm\") pod \"d0ba2aea-3426-4deb-937b-fe0954cb0372\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") "
Jul 21 17:54:14.172417 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.172296    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0ba2aea-3426-4deb-937b-fe0954cb0372-cabundle-cert\") pod \"d0ba2aea-3426-4deb-937b-fe0954cb0372\" (UID: \"d0ba2aea-3426-4deb-937b-fe0954cb0372\") "
Jul 21 17:54:14.172711 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.172679    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ba2aea-3426-4deb-937b-fe0954cb0372-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0ba2aea-3426-4deb-937b-fe0954cb0372" (UID: "d0ba2aea-3426-4deb-937b-fe0954cb0372"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:54:14.172788 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.172726    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ba2aea-3426-4deb-937b-fe0954cb0372-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0ba2aea-3426-4deb-937b-fe0954cb0372" (UID: "d0ba2aea-3426-4deb-937b-fe0954cb0372"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:54:14.173192 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.173165    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ba2aea-3426-4deb-937b-fe0954cb0372-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0ba2aea-3426-4deb-937b-fe0954cb0372" (UID: "d0ba2aea-3426-4deb-937b-fe0954cb0372"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:54:14.174547 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.174519    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ba2aea-3426-4deb-937b-fe0954cb0372-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0ba2aea-3426-4deb-937b-fe0954cb0372" (UID: "d0ba2aea-3426-4deb-937b-fe0954cb0372"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:54:14.174729 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.174705    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ba2aea-3426-4deb-937b-fe0954cb0372-kube-api-access-8c9bm" (OuterVolumeSpecName: "kube-api-access-8c9bm") pod "d0ba2aea-3426-4deb-937b-fe0954cb0372" (UID: "d0ba2aea-3426-4deb-937b-fe0954cb0372"). InnerVolumeSpecName "kube-api-access-8c9bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:54:14.273359 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.273258    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0ba2aea-3426-4deb-937b-fe0954cb0372-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:54:14.273359 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.273293    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0ba2aea-3426-4deb-937b-fe0954cb0372-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:54:14.273359 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.273308    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0ba2aea-3426-4deb-937b-fe0954cb0372-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:54:14.273359 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.273321    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0ba2aea-3426-4deb-937b-fe0954cb0372-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:54:14.273359 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:14.273332    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c9bm\" (UniqueName: \"kubernetes.io/projected/d0ba2aea-3426-4deb-937b-fe0954cb0372-kube-api-access-8c9bm\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:54:15.052487 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:15.052457    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"
Jul 21 17:54:15.052918 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:15.052451    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g" event={"ID":"d0ba2aea-3426-4deb-937b-fe0954cb0372","Type":"ContainerDied","Data":"1a9d8e12ad47c633d2f054447e27ac9125d7747a2e6dc3584aaa1ff9dea09136"}
Jul 21 17:54:15.052918 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:15.052600    2562 scope.go:117] "RemoveContainer" containerID="99dc80fe36ef97122c0032e477e75f618e5cef27b25cfb565a29ed10e9718e04"
Jul 21 17:54:15.061213 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:15.061194    2562 scope.go:117] "RemoveContainer" containerID="77c0708404cd6688d75234661fe13903cb5fecd6c865c0aa0231ce26ec430880"
Jul 21 17:54:15.068706 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:15.068690    2562 scope.go:117] "RemoveContainer" containerID="04ca3909846f1449ed48e710477b06ad51bb175458c3579d71c45d063bb38430"
Jul 21 17:54:15.072897 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:15.072786    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"]
Jul 21 17:54:15.074730 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:15.074710    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m4k4g"]
Jul 21 17:54:16.921515 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:16.921482    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0ba2aea-3426-4deb-937b-fe0954cb0372" path="/var/lib/kubelet/pods/d0ba2aea-3426-4deb-937b-fe0954cb0372/volumes"
Jul 21 17:54:17.533266 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.533235    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"]
Jul 21 17:54:17.533560 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.533547    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerName="storage-initializer"
Jul 21 17:54:17.533606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.533562    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerName="storage-initializer"
Jul 21 17:54:17.533606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.533569    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerName="kserve-container"
Jul 21 17:54:17.533606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.533575    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerName="kserve-container"
Jul 21 17:54:17.533606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.533587    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerName="kube-rbac-proxy"
Jul 21 17:54:17.533606 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.533593    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerName="kube-rbac-proxy"
Jul 21 17:54:17.533760 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.533646    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerName="kserve-container"
Jul 21 17:54:17.533760 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.533653    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0ba2aea-3426-4deb-937b-fe0954cb0372" containerName="kube-rbac-proxy"
Jul 21 17:54:17.538742 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.538725    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.541513 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.541490    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 17:54:17.541694 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.541568    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 17:54:17.541694 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.541649    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 17:54:17.541825 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.541799    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 17:54:17.542843 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.542823    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 17:54:17.542964 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.542823    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Jul 21 17:54:17.542964 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.542855    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Jul 21 17:54:17.547033 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.547011    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"]
Jul 21 17:54:17.600514 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.600478    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dff10611-bff3-4930-aec9-9d896c6b02b0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.600701 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.600522    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff10611-bff3-4930-aec9-9d896c6b02b0-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.600701 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.600605    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dff10611-bff3-4930-aec9-9d896c6b02b0-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.600701 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.600637    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsndd\" (UniqueName: \"kubernetes.io/projected/dff10611-bff3-4930-aec9-9d896c6b02b0-kube-api-access-xsndd\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.600701 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.600669    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff10611-bff3-4930-aec9-9d896c6b02b0-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.701198 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.701170    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dff10611-bff3-4930-aec9-9d896c6b02b0-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.701351 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.701215    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsndd\" (UniqueName: \"kubernetes.io/projected/dff10611-bff3-4930-aec9-9d896c6b02b0-kube-api-access-xsndd\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.701351 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.701320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff10611-bff3-4930-aec9-9d896c6b02b0-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.701451 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.701356    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dff10611-bff3-4930-aec9-9d896c6b02b0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.701451 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.701379    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff10611-bff3-4930-aec9-9d896c6b02b0-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.701743 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.701724    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff10611-bff3-4930-aec9-9d896c6b02b0-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.701892 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.701871    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dff10611-bff3-4930-aec9-9d896c6b02b0-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.702055 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.702036    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dff10611-bff3-4930-aec9-9d896c6b02b0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.703974 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.703954    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff10611-bff3-4930-aec9-9d896c6b02b0-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.709180 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.709161    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsndd\" (UniqueName: \"kubernetes.io/projected/dff10611-bff3-4930-aec9-9d896c6b02b0-kube-api-access-xsndd\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.849893 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.849796    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:17.988254 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:17.988230    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"]
Jul 21 17:54:17.990638 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:54:17.990605    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff10611_bff3_4930_aec9_9d896c6b02b0.slice/crio-c301a8ac750c990ae000654eed3f5050f13474f4fd1f0d5ab2950b04f6cb862b WatchSource:0}: Error finding container c301a8ac750c990ae000654eed3f5050f13474f4fd1f0d5ab2950b04f6cb862b: Status 404 returned error can't find the container with id c301a8ac750c990ae000654eed3f5050f13474f4fd1f0d5ab2950b04f6cb862b
Jul 21 17:54:18.067417 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:18.067381    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7" event={"ID":"dff10611-bff3-4930-aec9-9d896c6b02b0","Type":"ContainerStarted","Data":"7ddd9eb10ddcd912a3fd6697fbdb22e1ad431057859553f6b6f648d07f93687e"}
Jul 21 17:54:18.067548 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:18.067442    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7" event={"ID":"dff10611-bff3-4930-aec9-9d896c6b02b0","Type":"ContainerStarted","Data":"c301a8ac750c990ae000654eed3f5050f13474f4fd1f0d5ab2950b04f6cb862b"}
Jul 21 17:54:23.090225 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:23.090186    2562 generic.go:356] "Generic (PLEG): container finished" podID="dff10611-bff3-4930-aec9-9d896c6b02b0" containerID="7ddd9eb10ddcd912a3fd6697fbdb22e1ad431057859553f6b6f648d07f93687e" exitCode=0
Jul 21 17:54:23.090644 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:23.090259    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7" event={"ID":"dff10611-bff3-4930-aec9-9d896c6b02b0","Type":"ContainerDied","Data":"7ddd9eb10ddcd912a3fd6697fbdb22e1ad431057859553f6b6f648d07f93687e"}
Jul 21 17:54:24.097796 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:24.097763    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7" event={"ID":"dff10611-bff3-4930-aec9-9d896c6b02b0","Type":"ContainerStarted","Data":"12e413ae40a399e28c51b2f6d64806ecdfaba8465d812cb3105c63aa981493fb"}
Jul 21 17:54:24.098187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:24.097827    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7" event={"ID":"dff10611-bff3-4930-aec9-9d896c6b02b0","Type":"ContainerStarted","Data":"69d4b596ce1e27b75e2095945de630d39c6fef72a65f5684da04496ba9c24c4e"}
Jul 21 17:54:24.098187 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:24.098097    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:24.098262 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:24.098240    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:24.099705 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:24.099677    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7" podUID="dff10611-bff3-4930-aec9-9d896c6b02b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 21 17:54:24.118417 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:24.118370    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7" podStartSLOduration=2.748299425 podStartE2EDuration="7.118353967s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.370054542s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:54:24.117337871 +0000 UTC m=+1715.688959657" watchObservedRunningTime="2026-07-21 17:54:24.118353967 +0000 UTC m=+1715.689975724"
Jul 21 17:54:25.102755 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:25.102719    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7" podUID="dff10611-bff3-4930-aec9-9d896c6b02b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 21 17:54:30.106743 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:30.106716    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:30.107237 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:30.107208    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7" podUID="dff10611-bff3-4930-aec9-9d896c6b02b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 21 17:54:40.108274 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:40.108240    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:52.705922 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:52.705876    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"]
Jul 21 17:54:52.706501 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:52.706180    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7" podUID="dff10611-bff3-4930-aec9-9d896c6b02b0" containerName="kube-rbac-proxy" containerID="cri-o://12e413ae40a399e28c51b2f6d64806ecdfaba8465d812cb3105c63aa981493fb" gracePeriod=30
Jul 21 17:54:52.706501 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:52.706216    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7" podUID="dff10611-bff3-4930-aec9-9d896c6b02b0" containerName="kserve-container" containerID="cri-o://69d4b596ce1e27b75e2095945de630d39c6fef72a65f5684da04496ba9c24c4e" gracePeriod=30
Jul 21 17:54:53.227296 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:53.227257    2562 generic.go:356] "Generic (PLEG): container finished" podID="dff10611-bff3-4930-aec9-9d896c6b02b0" containerID="12e413ae40a399e28c51b2f6d64806ecdfaba8465d812cb3105c63aa981493fb" exitCode=2
Jul 21 17:54:53.227464 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:53.227332    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7" event={"ID":"dff10611-bff3-4930-aec9-9d896c6b02b0","Type":"ContainerDied","Data":"12e413ae40a399e28c51b2f6d64806ecdfaba8465d812cb3105c63aa981493fb"}
Jul 21 17:54:54.060658 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.060631    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:54.085114 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.085080    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dff10611-bff3-4930-aec9-9d896c6b02b0-kube-rbac-proxy-sar-config\") pod \"dff10611-bff3-4930-aec9-9d896c6b02b0\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") "
Jul 21 17:54:54.085266 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.085142    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff10611-bff3-4930-aec9-9d896c6b02b0-kserve-provision-location\") pod \"dff10611-bff3-4930-aec9-9d896c6b02b0\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") "
Jul 21 17:54:54.085266 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.085176    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dff10611-bff3-4930-aec9-9d896c6b02b0-cabundle-cert\") pod \"dff10611-bff3-4930-aec9-9d896c6b02b0\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") "
Jul 21 17:54:54.085266 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.085209    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsndd\" (UniqueName: \"kubernetes.io/projected/dff10611-bff3-4930-aec9-9d896c6b02b0-kube-api-access-xsndd\") pod \"dff10611-bff3-4930-aec9-9d896c6b02b0\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") "
Jul 21 17:54:54.085266 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.085241    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff10611-bff3-4930-aec9-9d896c6b02b0-proxy-tls\") pod \"dff10611-bff3-4930-aec9-9d896c6b02b0\" (UID: \"dff10611-bff3-4930-aec9-9d896c6b02b0\") "
Jul 21 17:54:54.085475 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.085450    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff10611-bff3-4930-aec9-9d896c6b02b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dff10611-bff3-4930-aec9-9d896c6b02b0" (UID: "dff10611-bff3-4930-aec9-9d896c6b02b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:54:54.085530 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.085511    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff10611-bff3-4930-aec9-9d896c6b02b0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dff10611-bff3-4930-aec9-9d896c6b02b0" (UID: "dff10611-bff3-4930-aec9-9d896c6b02b0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:54:54.085578 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.085542    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dff10611-bff3-4930-aec9-9d896c6b02b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:54:54.086211 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.086189    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff10611-bff3-4930-aec9-9d896c6b02b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dff10611-bff3-4930-aec9-9d896c6b02b0" (UID: "dff10611-bff3-4930-aec9-9d896c6b02b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:54:54.087593 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.087572    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dff10611-bff3-4930-aec9-9d896c6b02b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dff10611-bff3-4930-aec9-9d896c6b02b0" (UID: "dff10611-bff3-4930-aec9-9d896c6b02b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:54:54.087719 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.087702    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff10611-bff3-4930-aec9-9d896c6b02b0-kube-api-access-xsndd" (OuterVolumeSpecName: "kube-api-access-xsndd") pod "dff10611-bff3-4930-aec9-9d896c6b02b0" (UID: "dff10611-bff3-4930-aec9-9d896c6b02b0"). InnerVolumeSpecName "kube-api-access-xsndd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:54:54.186856 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.186829    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dff10611-bff3-4930-aec9-9d896c6b02b0-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:54:54.186856 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.186857    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff10611-bff3-4930-aec9-9d896c6b02b0-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:54:54.187038 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.186868    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dff10611-bff3-4930-aec9-9d896c6b02b0-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:54:54.187038 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.186876    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsndd\" (UniqueName: \"kubernetes.io/projected/dff10611-bff3-4930-aec9-9d896c6b02b0-kube-api-access-xsndd\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:54:54.239775 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.239739    2562 generic.go:356] "Generic (PLEG): container finished" podID="dff10611-bff3-4930-aec9-9d896c6b02b0" containerID="69d4b596ce1e27b75e2095945de630d39c6fef72a65f5684da04496ba9c24c4e" exitCode=0
Jul 21 17:54:54.239961 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.239805    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"
Jul 21 17:54:54.239961 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.239829    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7" event={"ID":"dff10611-bff3-4930-aec9-9d896c6b02b0","Type":"ContainerDied","Data":"69d4b596ce1e27b75e2095945de630d39c6fef72a65f5684da04496ba9c24c4e"}
Jul 21 17:54:54.239961 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.239911    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7" event={"ID":"dff10611-bff3-4930-aec9-9d896c6b02b0","Type":"ContainerDied","Data":"c301a8ac750c990ae000654eed3f5050f13474f4fd1f0d5ab2950b04f6cb862b"}
Jul 21 17:54:54.239961 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.239936    2562 scope.go:117] "RemoveContainer" containerID="12e413ae40a399e28c51b2f6d64806ecdfaba8465d812cb3105c63aa981493fb"
Jul 21 17:54:54.248622 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.248605    2562 scope.go:117] "RemoveContainer" containerID="69d4b596ce1e27b75e2095945de630d39c6fef72a65f5684da04496ba9c24c4e"
Jul 21 17:54:54.258592 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.258550    2562 scope.go:117] "RemoveContainer" containerID="7ddd9eb10ddcd912a3fd6697fbdb22e1ad431057859553f6b6f648d07f93687e"
Jul 21 17:54:54.262889 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.262865    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"]
Jul 21 17:54:54.267114 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.266974    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-j4ms7"]
Jul 21 17:54:54.267190 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.267141    2562 scope.go:117] "RemoveContainer" containerID="12e413ae40a399e28c51b2f6d64806ecdfaba8465d812cb3105c63aa981493fb"
Jul 21 17:54:54.267438 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:54:54.267419    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e413ae40a399e28c51b2f6d64806ecdfaba8465d812cb3105c63aa981493fb\": container with ID starting with 12e413ae40a399e28c51b2f6d64806ecdfaba8465d812cb3105c63aa981493fb not found: ID does not exist" containerID="12e413ae40a399e28c51b2f6d64806ecdfaba8465d812cb3105c63aa981493fb"
Jul 21 17:54:54.267475 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.267447    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e413ae40a399e28c51b2f6d64806ecdfaba8465d812cb3105c63aa981493fb"} err="failed to get container status \"12e413ae40a399e28c51b2f6d64806ecdfaba8465d812cb3105c63aa981493fb\": rpc error: code = NotFound desc = could not find container \"12e413ae40a399e28c51b2f6d64806ecdfaba8465d812cb3105c63aa981493fb\": container with ID starting with 12e413ae40a399e28c51b2f6d64806ecdfaba8465d812cb3105c63aa981493fb not found: ID does not exist"
Jul 21 17:54:54.267475 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.267466    2562 scope.go:117] "RemoveContainer" containerID="69d4b596ce1e27b75e2095945de630d39c6fef72a65f5684da04496ba9c24c4e"
Jul 21 17:54:54.267742 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:54:54.267722    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d4b596ce1e27b75e2095945de630d39c6fef72a65f5684da04496ba9c24c4e\": container with ID starting with 69d4b596ce1e27b75e2095945de630d39c6fef72a65f5684da04496ba9c24c4e not found: ID does not exist" containerID="69d4b596ce1e27b75e2095945de630d39c6fef72a65f5684da04496ba9c24c4e"
Jul 21 17:54:54.267789 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.267748    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d4b596ce1e27b75e2095945de630d39c6fef72a65f5684da04496ba9c24c4e"} err="failed to get container status \"69d4b596ce1e27b75e2095945de630d39c6fef72a65f5684da04496ba9c24c4e\": rpc error: code = NotFound desc = could not find container \"69d4b596ce1e27b75e2095945de630d39c6fef72a65f5684da04496ba9c24c4e\": container with ID starting with 69d4b596ce1e27b75e2095945de630d39c6fef72a65f5684da04496ba9c24c4e not found: ID does not exist"
Jul 21 17:54:54.267789 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.267764    2562 scope.go:117] "RemoveContainer" containerID="7ddd9eb10ddcd912a3fd6697fbdb22e1ad431057859553f6b6f648d07f93687e"
Jul 21 17:54:54.267980 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:54:54.267961    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ddd9eb10ddcd912a3fd6697fbdb22e1ad431057859553f6b6f648d07f93687e\": container with ID starting with 7ddd9eb10ddcd912a3fd6697fbdb22e1ad431057859553f6b6f648d07f93687e not found: ID does not exist" containerID="7ddd9eb10ddcd912a3fd6697fbdb22e1ad431057859553f6b6f648d07f93687e"
Jul 21 17:54:54.268033 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.267990    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ddd9eb10ddcd912a3fd6697fbdb22e1ad431057859553f6b6f648d07f93687e"} err="failed to get container status \"7ddd9eb10ddcd912a3fd6697fbdb22e1ad431057859553f6b6f648d07f93687e\": rpc error: code = NotFound desc = could not find container \"7ddd9eb10ddcd912a3fd6697fbdb22e1ad431057859553f6b6f648d07f93687e\": container with ID starting with 7ddd9eb10ddcd912a3fd6697fbdb22e1ad431057859553f6b6f648d07f93687e not found: ID does not exist"
Jul 21 17:54:54.921677 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:54.921645    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dff10611-bff3-4930-aec9-9d896c6b02b0" path="/var/lib/kubelet/pods/dff10611-bff3-4930-aec9-9d896c6b02b0/volumes"
Jul 21 17:54:57.865711 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.865677    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"]
Jul 21 17:54:57.866172 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.865982    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff10611-bff3-4930-aec9-9d896c6b02b0" containerName="storage-initializer"
Jul 21 17:54:57.866172 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.865992    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff10611-bff3-4930-aec9-9d896c6b02b0" containerName="storage-initializer"
Jul 21 17:54:57.866172 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.866000    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff10611-bff3-4930-aec9-9d896c6b02b0" containerName="kserve-container"
Jul 21 17:54:57.866172 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.866006    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff10611-bff3-4930-aec9-9d896c6b02b0" containerName="kserve-container"
Jul 21 17:54:57.866172 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.866017    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff10611-bff3-4930-aec9-9d896c6b02b0" containerName="kube-rbac-proxy"
Jul 21 17:54:57.866172 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.866049    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff10611-bff3-4930-aec9-9d896c6b02b0" containerName="kube-rbac-proxy"
Jul 21 17:54:57.866172 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.866174    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff10611-bff3-4930-aec9-9d896c6b02b0" containerName="kserve-container"
Jul 21 17:54:57.866505 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.866187    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff10611-bff3-4930-aec9-9d896c6b02b0" containerName="kube-rbac-proxy"
Jul 21 17:54:57.870971 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.870952    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:57.873946 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.873926    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 17:54:57.874059 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.873930    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 17:54:57.874059 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.874044    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 17:54:57.874196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.874058    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jul 21 17:54:57.874196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.874058    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jul 21 17:54:57.874196 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.874094    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 17:54:57.875036 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.875019    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 17:54:57.880267 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.880247    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"]
Jul 21 17:54:57.915095 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.915047    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-lzlql\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:57.915220 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.915136    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3f253dc-9500-4df3-91ad-4b2623b3c96a-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-lzlql\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:57.915220 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.915165    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-lzlql\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:57.915220 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.915188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssnlt\" (UniqueName: \"kubernetes.io/projected/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kube-api-access-ssnlt\") pod \"isvc-lightgbm-predictor-54f85ddf88-lzlql\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:57.915380 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:57.915229    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3f253dc-9500-4df3-91ad-4b2623b3c96a-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-lzlql\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:58.015958 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:58.015916    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-lzlql\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:58.016133 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:58.015966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3f253dc-9500-4df3-91ad-4b2623b3c96a-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-lzlql\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:58.016133 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:58.015994    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-lzlql\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:58.016133 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:58.016021    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssnlt\" (UniqueName: \"kubernetes.io/projected/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kube-api-access-ssnlt\") pod \"isvc-lightgbm-predictor-54f85ddf88-lzlql\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:58.016133 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:58.016049    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3f253dc-9500-4df3-91ad-4b2623b3c96a-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-lzlql\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:58.016364 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:54:58.016193    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-predictor-serving-cert: secret "isvc-lightgbm-predictor-serving-cert" not found
Jul 21 17:54:58.016364 ip-10-0-135-228 kubenswrapper[2562]: E0721 17:54:58.016262    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3f253dc-9500-4df3-91ad-4b2623b3c96a-proxy-tls podName:d3f253dc-9500-4df3-91ad-4b2623b3c96a nodeName:}" failed. No retries permitted until 2026-07-21 17:54:58.516241245 +0000 UTC m=+1750.087862981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d3f253dc-9500-4df3-91ad-4b2623b3c96a-proxy-tls") pod "isvc-lightgbm-predictor-54f85ddf88-lzlql" (UID: "d3f253dc-9500-4df3-91ad-4b2623b3c96a") : secret "isvc-lightgbm-predictor-serving-cert" not found
Jul 21 17:54:58.016638 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:58.016617    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-lzlql\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:58.016708 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:58.016675    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-lzlql\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:58.016921 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:58.016904    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3f253dc-9500-4df3-91ad-4b2623b3c96a-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-lzlql\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:58.024683 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:58.024664    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssnlt\" (UniqueName: \"kubernetes.io/projected/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kube-api-access-ssnlt\") pod \"isvc-lightgbm-predictor-54f85ddf88-lzlql\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:58.519442 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:58.519408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3f253dc-9500-4df3-91ad-4b2623b3c96a-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-lzlql\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:58.521854 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:58.521835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3f253dc-9500-4df3-91ad-4b2623b3c96a-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-lzlql\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:58.782531 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:58.782436    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:54:58.919238 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:54:58.919197    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f253dc_9500_4df3_91ad_4b2623b3c96a.slice/crio-b50af1d44345d933a533f4d6fb2a6d02353cd64997fc641e3e65d6c106edb9ba WatchSource:0}: Error finding container b50af1d44345d933a533f4d6fb2a6d02353cd64997fc641e3e65d6c106edb9ba: Status 404 returned error can't find the container with id b50af1d44345d933a533f4d6fb2a6d02353cd64997fc641e3e65d6c106edb9ba
Jul 21 17:54:58.921949 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:58.921929    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"]
Jul 21 17:54:59.263041 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:59.263001    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" event={"ID":"d3f253dc-9500-4df3-91ad-4b2623b3c96a","Type":"ContainerStarted","Data":"63b70fb0a6740e646ff21e096c1e3e60c6843e5ab23c44d644117b564176fafd"}
Jul 21 17:54:59.263228 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:54:59.263078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" event={"ID":"d3f253dc-9500-4df3-91ad-4b2623b3c96a","Type":"ContainerStarted","Data":"b50af1d44345d933a533f4d6fb2a6d02353cd64997fc641e3e65d6c106edb9ba"}
Jul 21 17:55:03.283101 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:03.283046    2562 generic.go:356] "Generic (PLEG): container finished" podID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerID="63b70fb0a6740e646ff21e096c1e3e60c6843e5ab23c44d644117b564176fafd" exitCode=0
Jul 21 17:55:03.283473 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:03.283123    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" event={"ID":"d3f253dc-9500-4df3-91ad-4b2623b3c96a","Type":"ContainerDied","Data":"63b70fb0a6740e646ff21e096c1e3e60c6843e5ab23c44d644117b564176fafd"}
Jul 21 17:55:03.284251 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:03.284234    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 17:55:10.319551 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:10.319513    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" event={"ID":"d3f253dc-9500-4df3-91ad-4b2623b3c96a","Type":"ContainerStarted","Data":"e4e905c7d0fcc15099ebab189d3f8914de9d2d5ae169d528db8bfb0a521793db"}
Jul 21 17:55:10.320028 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:10.319582    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" event={"ID":"d3f253dc-9500-4df3-91ad-4b2623b3c96a","Type":"ContainerStarted","Data":"ad26e6bcb307de8d4101cb3fc212e28ef31f1338298347b44489df9dd5b88ffc"}
Jul 21 17:55:10.320028 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:10.319787    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:55:10.340521 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:10.340469    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" podStartSLOduration=3.659726154 podStartE2EDuration="13.340448555s" totalImagesPullingTime="6.194460803s" totalInitContainerRuntime="3.486261598s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:55:10.33929161 +0000 UTC m=+1761.910913446" watchObservedRunningTime="2026-07-21 17:55:10.340448555 +0000 UTC m=+1761.912070311"
Jul 21 17:55:11.324274 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:11.324242    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:55:11.325633 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:11.325603    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 21 17:55:12.327774 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:12.327732    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 21 17:55:17.332467 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:17.332433    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:55:17.332891 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:17.332866    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 21 17:55:27.333774 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:27.333732    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 21 17:55:37.333378 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:37.333338    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 21 17:55:47.333652 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:47.333610    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 21 17:55:48.943343 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:48.943316    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 17:55:48.949614 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:48.949593    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 17:55:57.333001 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:55:57.332959    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 21 17:56:07.333464 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:07.333423    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 21 17:56:17.333867 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:17.333827    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 21 17:56:27.333996 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:27.333969    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:56:28.007355 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:28.007325    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"]
Jul 21 17:56:28.007625 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:28.007594    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kserve-container" containerID="cri-o://ad26e6bcb307de8d4101cb3fc212e28ef31f1338298347b44489df9dd5b88ffc" gracePeriod=30
Jul 21 17:56:28.007775 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:28.007685    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kube-rbac-proxy" containerID="cri-o://e4e905c7d0fcc15099ebab189d3f8914de9d2d5ae169d528db8bfb0a521793db" gracePeriod=30
Jul 21 17:56:28.653716 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:28.653683    2562 generic.go:356] "Generic (PLEG): container finished" podID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerID="e4e905c7d0fcc15099ebab189d3f8914de9d2d5ae169d528db8bfb0a521793db" exitCode=2
Jul 21 17:56:28.654135 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:28.653726    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" event={"ID":"d3f253dc-9500-4df3-91ad-4b2623b3c96a","Type":"ContainerDied","Data":"e4e905c7d0fcc15099ebab189d3f8914de9d2d5ae169d528db8bfb0a521793db"}
Jul 21 17:56:32.328574 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:32.328535    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 21 17:56:33.676709 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.676679    2562 generic.go:356] "Generic (PLEG): container finished" podID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerID="ad26e6bcb307de8d4101cb3fc212e28ef31f1338298347b44489df9dd5b88ffc" exitCode=0
Jul 21 17:56:33.677111 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.676729    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" event={"ID":"d3f253dc-9500-4df3-91ad-4b2623b3c96a","Type":"ContainerDied","Data":"ad26e6bcb307de8d4101cb3fc212e28ef31f1338298347b44489df9dd5b88ffc"}
Jul 21 17:56:33.768698 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.768674    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:56:33.887786 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.887694    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssnlt\" (UniqueName: \"kubernetes.io/projected/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kube-api-access-ssnlt\") pod \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") "
Jul 21 17:56:33.887786 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.887738    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kserve-provision-location\") pod \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") "
Jul 21 17:56:33.888020 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.887787    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3f253dc-9500-4df3-91ad-4b2623b3c96a-cabundle-cert\") pod \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") "
Jul 21 17:56:33.888020 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.887852    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3f253dc-9500-4df3-91ad-4b2623b3c96a-proxy-tls\") pod \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") "
Jul 21 17:56:33.888020 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.887945    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kube-rbac-proxy-sar-config\") pod \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\" (UID: \"d3f253dc-9500-4df3-91ad-4b2623b3c96a\") "
Jul 21 17:56:33.888215 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.888177    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3f253dc-9500-4df3-91ad-4b2623b3c96a" (UID: "d3f253dc-9500-4df3-91ad-4b2623b3c96a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:56:33.888215 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.888196    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f253dc-9500-4df3-91ad-4b2623b3c96a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3f253dc-9500-4df3-91ad-4b2623b3c96a" (UID: "d3f253dc-9500-4df3-91ad-4b2623b3c96a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:56:33.888415 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.888381    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d3f253dc-9500-4df3-91ad-4b2623b3c96a" (UID: "d3f253dc-9500-4df3-91ad-4b2623b3c96a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:56:33.890180 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.890150    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kube-api-access-ssnlt" (OuterVolumeSpecName: "kube-api-access-ssnlt") pod "d3f253dc-9500-4df3-91ad-4b2623b3c96a" (UID: "d3f253dc-9500-4df3-91ad-4b2623b3c96a"). InnerVolumeSpecName "kube-api-access-ssnlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:56:33.890271 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.890172    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f253dc-9500-4df3-91ad-4b2623b3c96a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d3f253dc-9500-4df3-91ad-4b2623b3c96a" (UID: "d3f253dc-9500-4df3-91ad-4b2623b3c96a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:56:33.989094 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.989034    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:56:33.989094 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.989092    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssnlt\" (UniqueName: \"kubernetes.io/projected/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kube-api-access-ssnlt\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:56:33.989322 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.989109    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3f253dc-9500-4df3-91ad-4b2623b3c96a-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:56:33.989322 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.989119    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3f253dc-9500-4df3-91ad-4b2623b3c96a-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:56:33.989322 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:33.989128    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3f253dc-9500-4df3-91ad-4b2623b3c96a-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 17:56:34.682990 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:34.682948    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql" event={"ID":"d3f253dc-9500-4df3-91ad-4b2623b3c96a","Type":"ContainerDied","Data":"b50af1d44345d933a533f4d6fb2a6d02353cd64997fc641e3e65d6c106edb9ba"}
Jul 21 17:56:34.683428 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:34.683000    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"
Jul 21 17:56:34.683428 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:34.683033    2562 scope.go:117] "RemoveContainer" containerID="e4e905c7d0fcc15099ebab189d3f8914de9d2d5ae169d528db8bfb0a521793db"
Jul 21 17:56:34.692004 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:34.691986    2562 scope.go:117] "RemoveContainer" containerID="ad26e6bcb307de8d4101cb3fc212e28ef31f1338298347b44489df9dd5b88ffc"
Jul 21 17:56:34.699607 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:34.699588    2562 scope.go:117] "RemoveContainer" containerID="63b70fb0a6740e646ff21e096c1e3e60c6843e5ab23c44d644117b564176fafd"
Jul 21 17:56:34.707477 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:34.707449    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"]
Jul 21 17:56:34.709958 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:34.709933    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-lzlql"]
Jul 21 17:56:34.921696 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:56:34.921652    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" path="/var/lib/kubelet/pods/d3f253dc-9500-4df3-91ad-4b2623b3c96a/volumes"
Jul 21 17:58:09.584001 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.583962    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"]
Jul 21 17:58:09.584672 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.584459    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="storage-initializer"
Jul 21 17:58:09.584672 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.584486    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="storage-initializer"
Jul 21 17:58:09.584672 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.584518    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kserve-container"
Jul 21 17:58:09.584672 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.584528    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kserve-container"
Jul 21 17:58:09.584672 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.584545    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kube-rbac-proxy"
Jul 21 17:58:09.584672 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.584553    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kube-rbac-proxy"
Jul 21 17:58:09.584672 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.584631    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kserve-container"
Jul 21 17:58:09.584672 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.584647    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3f253dc-9500-4df3-91ad-4b2623b3c96a" containerName="kube-rbac-proxy"
Jul 21 17:58:09.587588 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.587567    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:09.590357 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.590336    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 17:58:09.590489 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.590341    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 21 17:58:09.591502 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.591445    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 17:58:09.591502 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.591470    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 17:58:09.591502 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.591491    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 21 17:58:09.591722 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.591525    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 17:58:09.591722 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.591497    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 17:58:09.598162 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.598139    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"]
Jul 21 17:58:09.694913 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.694872    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8z4n\" (UniqueName: \"kubernetes.io/projected/718776d3-4895-408e-891c-54f816e08ed6-kube-api-access-k8z4n\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:09.694913 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.694911    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/718776d3-4895-408e-891c-54f816e08ed6-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:09.695216 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.694937    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/718776d3-4895-408e-891c-54f816e08ed6-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:09.695216 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.694952    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/718776d3-4895-408e-891c-54f816e08ed6-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:09.695216 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.695018    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/718776d3-4895-408e-891c-54f816e08ed6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:09.795447 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.795414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/718776d3-4895-408e-891c-54f816e08ed6-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:09.795447 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.795444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/718776d3-4895-408e-891c-54f816e08ed6-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:09.795689 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.795479    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/718776d3-4895-408e-891c-54f816e08ed6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:09.795689 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.795524    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8z4n\" (UniqueName: \"kubernetes.io/projected/718776d3-4895-408e-891c-54f816e08ed6-kube-api-access-k8z4n\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:09.795689 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.795544    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/718776d3-4895-408e-891c-54f816e08ed6-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:09.795848 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.795817    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/718776d3-4895-408e-891c-54f816e08ed6-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:09.796214 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.796196    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/718776d3-4895-408e-891c-54f816e08ed6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:09.796273 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.796243    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/718776d3-4895-408e-891c-54f816e08ed6-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:09.798036 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.798014    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/718776d3-4895-408e-891c-54f816e08ed6-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:09.805011 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.804987    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8z4n\" (UniqueName: \"kubernetes.io/projected/718776d3-4895-408e-891c-54f816e08ed6-kube-api-access-k8z4n\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:09.898328 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:09.898237    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 17:58:10.033958 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:10.033934    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"]
Jul 21 17:58:10.036548 ip-10-0-135-228 kubenswrapper[2562]: W0721 17:58:10.036518    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod718776d3_4895_408e_891c_54f816e08ed6.slice/crio-3ef7c65cfe2a6a4a3399152eea4b15306a82ba811dd12ab7be671788fe8f5243 WatchSource:0}: Error finding container 3ef7c65cfe2a6a4a3399152eea4b15306a82ba811dd12ab7be671788fe8f5243: Status 404 returned error can't find the container with id 3ef7c65cfe2a6a4a3399152eea4b15306a82ba811dd12ab7be671788fe8f5243
Jul 21 17:58:10.062822 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:10.062797    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6" event={"ID":"718776d3-4895-408e-891c-54f816e08ed6","Type":"ContainerStarted","Data":"3ef7c65cfe2a6a4a3399152eea4b15306a82ba811dd12ab7be671788fe8f5243"}
Jul 21 17:58:11.067849 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:11.067816    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6" event={"ID":"718776d3-4895-408e-891c-54f816e08ed6","Type":"ContainerStarted","Data":"c3149826956ac399547fa98f7a16f7f10ede25ccb59faf90c3dd665c4dab96cb"}
Jul 21 17:58:15.084854 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:15.084818    2562 generic.go:356] "Generic (PLEG): container finished" podID="718776d3-4895-408e-891c-54f816e08ed6" containerID="c3149826956ac399547fa98f7a16f7f10ede25ccb59faf90c3dd665c4dab96cb" exitCode=0
Jul 21 17:58:15.085334 ip-10-0-135-228 kubenswrapper[2562]: I0721 17:58:15.084886    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6" event={"ID":"718776d3-4895-408e-891c-54f816e08ed6","Type":"ContainerDied","Data":"c3149826956ac399547fa98f7a16f7f10ede25ccb59faf90c3dd665c4dab96cb"}
Jul 21 18:00:27.720381 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:00:27.720346    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6" event={"ID":"718776d3-4895-408e-891c-54f816e08ed6","Type":"ContainerStarted","Data":"46300ca99c7189b69410f8e0f9817b23ee11157edf389741e24d5ae7e5061666"}
Jul 21 18:00:27.720846 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:00:27.720431    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6" event={"ID":"718776d3-4895-408e-891c-54f816e08ed6","Type":"ContainerStarted","Data":"fb7407c48d0d99cc892c1bc1de9a5cd368b2da26918e00487107a529e36506d2"}
Jul 21 18:00:27.720846 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:00:27.720494    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 18:00:27.751030 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:00:27.750975    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6" podStartSLOduration=3.220610638 podStartE2EDuration="2m18.75095894s" totalImagesPullingTime="2m11.552421642s" totalInitContainerRuntime="3.97792666s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:58:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:00:27.74984695 +0000 UTC m=+2079.321468730" watchObservedRunningTime="2026-07-21 18:00:27.75095894 +0000 UTC m=+2079.322580694"
Jul 21 18:00:28.725737 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:00:28.725698    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 18:00:34.734898 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:00:34.734868    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 18:00:48.984124 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:00:48.984092    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:00:48.984641 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:00:48.984099    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:01:04.739767 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:04.739736    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 18:01:09.757257 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:09.757178    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"]
Jul 21 18:01:09.757682 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:09.757436    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6" podUID="718776d3-4895-408e-891c-54f816e08ed6" containerName="kserve-container" containerID="cri-o://fb7407c48d0d99cc892c1bc1de9a5cd368b2da26918e00487107a529e36506d2" gracePeriod=30
Jul 21 18:01:09.757682 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:09.757517    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6" podUID="718776d3-4895-408e-891c-54f816e08ed6" containerName="kube-rbac-proxy" containerID="cri-o://46300ca99c7189b69410f8e0f9817b23ee11157edf389741e24d5ae7e5061666" gracePeriod=30
Jul 21 18:01:09.892562 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:09.892528    2562 generic.go:356] "Generic (PLEG): container finished" podID="718776d3-4895-408e-891c-54f816e08ed6" containerID="46300ca99c7189b69410f8e0f9817b23ee11157edf389741e24d5ae7e5061666" exitCode=2
Jul 21 18:01:09.892717 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:09.892585    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6" event={"ID":"718776d3-4895-408e-891c-54f816e08ed6","Type":"ContainerDied","Data":"46300ca99c7189b69410f8e0f9817b23ee11157edf389741e24d5ae7e5061666"}
Jul 21 18:01:10.800864 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.800841    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 18:01:10.888059 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.887960    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8z4n\" (UniqueName: \"kubernetes.io/projected/718776d3-4895-408e-891c-54f816e08ed6-kube-api-access-k8z4n\") pod \"718776d3-4895-408e-891c-54f816e08ed6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") "
Jul 21 18:01:10.888059 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.888041    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/718776d3-4895-408e-891c-54f816e08ed6-proxy-tls\") pod \"718776d3-4895-408e-891c-54f816e08ed6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") "
Jul 21 18:01:10.888059 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.888086    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/718776d3-4895-408e-891c-54f816e08ed6-cabundle-cert\") pod \"718776d3-4895-408e-891c-54f816e08ed6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") "
Jul 21 18:01:10.888378 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.888117    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/718776d3-4895-408e-891c-54f816e08ed6-kserve-provision-location\") pod \"718776d3-4895-408e-891c-54f816e08ed6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") "
Jul 21 18:01:10.888378 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.888261    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/718776d3-4895-408e-891c-54f816e08ed6-kube-rbac-proxy-sar-config\") pod \"718776d3-4895-408e-891c-54f816e08ed6\" (UID: \"718776d3-4895-408e-891c-54f816e08ed6\") "
Jul 21 18:01:10.888488 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.888403    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718776d3-4895-408e-891c-54f816e08ed6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "718776d3-4895-408e-891c-54f816e08ed6" (UID: "718776d3-4895-408e-891c-54f816e08ed6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:01:10.888488 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.888438    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718776d3-4895-408e-891c-54f816e08ed6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "718776d3-4895-408e-891c-54f816e08ed6" (UID: "718776d3-4895-408e-891c-54f816e08ed6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:01:10.888611 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.888593    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/718776d3-4895-408e-891c-54f816e08ed6-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:01:10.888652 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.888616    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/718776d3-4895-408e-891c-54f816e08ed6-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:01:10.888652 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.888598    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718776d3-4895-408e-891c-54f816e08ed6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "718776d3-4895-408e-891c-54f816e08ed6" (UID: "718776d3-4895-408e-891c-54f816e08ed6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:01:10.890868 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.890837    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718776d3-4895-408e-891c-54f816e08ed6-kube-api-access-k8z4n" (OuterVolumeSpecName: "kube-api-access-k8z4n") pod "718776d3-4895-408e-891c-54f816e08ed6" (UID: "718776d3-4895-408e-891c-54f816e08ed6"). InnerVolumeSpecName "kube-api-access-k8z4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:01:10.890868 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.890838    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/718776d3-4895-408e-891c-54f816e08ed6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "718776d3-4895-408e-891c-54f816e08ed6" (UID: "718776d3-4895-408e-891c-54f816e08ed6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:01:10.899092 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.899045    2562 generic.go:356] "Generic (PLEG): container finished" podID="718776d3-4895-408e-891c-54f816e08ed6" containerID="fb7407c48d0d99cc892c1bc1de9a5cd368b2da26918e00487107a529e36506d2" exitCode=0
Jul 21 18:01:10.899193 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.899110    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6" event={"ID":"718776d3-4895-408e-891c-54f816e08ed6","Type":"ContainerDied","Data":"fb7407c48d0d99cc892c1bc1de9a5cd368b2da26918e00487107a529e36506d2"}
Jul 21 18:01:10.899193 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.899136    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"
Jul 21 18:01:10.899193 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.899170    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6" event={"ID":"718776d3-4895-408e-891c-54f816e08ed6","Type":"ContainerDied","Data":"3ef7c65cfe2a6a4a3399152eea4b15306a82ba811dd12ab7be671788fe8f5243"}
Jul 21 18:01:10.899304 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.899196    2562 scope.go:117] "RemoveContainer" containerID="46300ca99c7189b69410f8e0f9817b23ee11157edf389741e24d5ae7e5061666"
Jul 21 18:01:10.907687 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.907671    2562 scope.go:117] "RemoveContainer" containerID="fb7407c48d0d99cc892c1bc1de9a5cd368b2da26918e00487107a529e36506d2"
Jul 21 18:01:10.915310 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.915285    2562 scope.go:117] "RemoveContainer" containerID="c3149826956ac399547fa98f7a16f7f10ede25ccb59faf90c3dd665c4dab96cb"
Jul 21 18:01:10.922340 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.922311    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"]
Jul 21 18:01:10.923845 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.923826    2562 scope.go:117] "RemoveContainer" containerID="46300ca99c7189b69410f8e0f9817b23ee11157edf389741e24d5ae7e5061666"
Jul 21 18:01:10.924146 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:01:10.924127    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46300ca99c7189b69410f8e0f9817b23ee11157edf389741e24d5ae7e5061666\": container with ID starting with 46300ca99c7189b69410f8e0f9817b23ee11157edf389741e24d5ae7e5061666 not found: ID does not exist" containerID="46300ca99c7189b69410f8e0f9817b23ee11157edf389741e24d5ae7e5061666"
Jul 21 18:01:10.924204 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.924153    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46300ca99c7189b69410f8e0f9817b23ee11157edf389741e24d5ae7e5061666"} err="failed to get container status \"46300ca99c7189b69410f8e0f9817b23ee11157edf389741e24d5ae7e5061666\": rpc error: code = NotFound desc = could not find container \"46300ca99c7189b69410f8e0f9817b23ee11157edf389741e24d5ae7e5061666\": container with ID starting with 46300ca99c7189b69410f8e0f9817b23ee11157edf389741e24d5ae7e5061666 not found: ID does not exist"
Jul 21 18:01:10.924204 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.924169    2562 scope.go:117] "RemoveContainer" containerID="fb7407c48d0d99cc892c1bc1de9a5cd368b2da26918e00487107a529e36506d2"
Jul 21 18:01:10.924413 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:01:10.924396    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7407c48d0d99cc892c1bc1de9a5cd368b2da26918e00487107a529e36506d2\": container with ID starting with fb7407c48d0d99cc892c1bc1de9a5cd368b2da26918e00487107a529e36506d2 not found: ID does not exist" containerID="fb7407c48d0d99cc892c1bc1de9a5cd368b2da26918e00487107a529e36506d2"
Jul 21 18:01:10.924481 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.924417    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7407c48d0d99cc892c1bc1de9a5cd368b2da26918e00487107a529e36506d2"} err="failed to get container status \"fb7407c48d0d99cc892c1bc1de9a5cd368b2da26918e00487107a529e36506d2\": rpc error: code = NotFound desc = could not find container \"fb7407c48d0d99cc892c1bc1de9a5cd368b2da26918e00487107a529e36506d2\": container with ID starting with fb7407c48d0d99cc892c1bc1de9a5cd368b2da26918e00487107a529e36506d2 not found: ID does not exist"
Jul 21 18:01:10.924481 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.924430    2562 scope.go:117] "RemoveContainer" containerID="c3149826956ac399547fa98f7a16f7f10ede25ccb59faf90c3dd665c4dab96cb"
Jul 21 18:01:10.924658 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:01:10.924642    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3149826956ac399547fa98f7a16f7f10ede25ccb59faf90c3dd665c4dab96cb\": container with ID starting with c3149826956ac399547fa98f7a16f7f10ede25ccb59faf90c3dd665c4dab96cb not found: ID does not exist" containerID="c3149826956ac399547fa98f7a16f7f10ede25ccb59faf90c3dd665c4dab96cb"
Jul 21 18:01:10.924702 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.924661    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3149826956ac399547fa98f7a16f7f10ede25ccb59faf90c3dd665c4dab96cb"} err="failed to get container status \"c3149826956ac399547fa98f7a16f7f10ede25ccb59faf90c3dd665c4dab96cb\": rpc error: code = NotFound desc = could not find container \"c3149826956ac399547fa98f7a16f7f10ede25ccb59faf90c3dd665c4dab96cb\": container with ID starting with c3149826956ac399547fa98f7a16f7f10ede25ccb59faf90c3dd665c4dab96cb not found: ID does not exist"
Jul 21 18:01:10.924840 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.924824    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wsfz6"]
Jul 21 18:01:10.989193 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.989159    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/718776d3-4895-408e-891c-54f816e08ed6-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:01:10.989193 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.989187    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/718776d3-4895-408e-891c-54f816e08ed6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:01:10.989193 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:10.989197    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8z4n\" (UniqueName: \"kubernetes.io/projected/718776d3-4895-408e-891c-54f816e08ed6-kube-api-access-k8z4n\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:01:12.920154 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:12.920126    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="718776d3-4895-408e-891c-54f816e08ed6" path="/var/lib/kubelet/pods/718776d3-4895-408e-891c-54f816e08ed6/volumes"
Jul 21 18:01:14.921575 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.921534    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"]
Jul 21 18:01:14.921989 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.921875    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="718776d3-4895-408e-891c-54f816e08ed6" containerName="storage-initializer"
Jul 21 18:01:14.921989 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.921890    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="718776d3-4895-408e-891c-54f816e08ed6" containerName="storage-initializer"
Jul 21 18:01:14.921989 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.921898    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="718776d3-4895-408e-891c-54f816e08ed6" containerName="kserve-container"
Jul 21 18:01:14.921989 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.921905    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="718776d3-4895-408e-891c-54f816e08ed6" containerName="kserve-container"
Jul 21 18:01:14.921989 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.921919    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="718776d3-4895-408e-891c-54f816e08ed6" containerName="kube-rbac-proxy"
Jul 21 18:01:14.921989 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.921925    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="718776d3-4895-408e-891c-54f816e08ed6" containerName="kube-rbac-proxy"
Jul 21 18:01:14.921989 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.921973    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="718776d3-4895-408e-891c-54f816e08ed6" containerName="kserve-container"
Jul 21 18:01:14.921989 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.921981    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="718776d3-4895-408e-891c-54f816e08ed6" containerName="kube-rbac-proxy"
Jul 21 18:01:14.925004 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.924980    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:14.928232 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.928210    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:01:14.928354 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.928277    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:01:14.928354 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.928214    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:01:14.928491 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.928353    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 21 18:01:14.929731 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.929713    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:01:14.929847 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.929710    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:01:14.929949 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.929932    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jul 21 18:01:14.931740 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:14.931711    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"]
Jul 21 18:01:15.023784 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.023744    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0336f600-50f6-4113-a741-a8969761127f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.023784 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.023781    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0336f600-50f6-4113-a741-a8969761127f-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.024093 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.023845    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw7lv\" (UniqueName: \"kubernetes.io/projected/0336f600-50f6-4113-a741-a8969761127f-kube-api-access-jw7lv\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.024093 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.023932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0336f600-50f6-4113-a741-a8969761127f-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.024225 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.024161    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0336f600-50f6-4113-a741-a8969761127f-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.124830 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.124788    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0336f600-50f6-4113-a741-a8969761127f-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.125025 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.124865    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0336f600-50f6-4113-a741-a8969761127f-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.125025 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.124907    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0336f600-50f6-4113-a741-a8969761127f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.125025 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.124931    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0336f600-50f6-4113-a741-a8969761127f-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.125025 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:01:15.124957    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-serving-cert: secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Jul 21 18:01:15.125025 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.124969    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw7lv\" (UniqueName: \"kubernetes.io/projected/0336f600-50f6-4113-a741-a8969761127f-kube-api-access-jw7lv\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.125326 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:01:15.125031    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0336f600-50f6-4113-a741-a8969761127f-proxy-tls podName:0336f600-50f6-4113-a741-a8969761127f nodeName:}" failed. No retries permitted until 2026-07-21 18:01:15.625010008 +0000 UTC m=+2127.196631740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0336f600-50f6-4113-a741-a8969761127f-proxy-tls") pod "isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz" (UID: "0336f600-50f6-4113-a741-a8969761127f") : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Jul 21 18:01:15.125391 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.125367    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0336f600-50f6-4113-a741-a8969761127f-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.125534 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.125515    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0336f600-50f6-4113-a741-a8969761127f-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.125627 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.125612    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0336f600-50f6-4113-a741-a8969761127f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.134115 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.134093    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw7lv\" (UniqueName: \"kubernetes.io/projected/0336f600-50f6-4113-a741-a8969761127f-kube-api-access-jw7lv\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.631079 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.631035    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0336f600-50f6-4113-a741-a8969761127f-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.633627 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.633606    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0336f600-50f6-4113-a741-a8969761127f-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.836488 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.836437    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:15.971477 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:15.971449    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"]
Jul 21 18:01:15.973500 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:01:15.973476    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0336f600_50f6_4113_a741_a8969761127f.slice/crio-22f76d2e270092f768da9c56c1a65239fa0b1b05fe26aa2b4c84193f3927efd7 WatchSource:0}: Error finding container 22f76d2e270092f768da9c56c1a65239fa0b1b05fe26aa2b4c84193f3927efd7: Status 404 returned error can't find the container with id 22f76d2e270092f768da9c56c1a65239fa0b1b05fe26aa2b4c84193f3927efd7
Jul 21 18:01:16.927834 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:16.927798    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz" event={"ID":"0336f600-50f6-4113-a741-a8969761127f","Type":"ContainerStarted","Data":"d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068"}
Jul 21 18:01:16.928021 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:16.927853    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz" event={"ID":"0336f600-50f6-4113-a741-a8969761127f","Type":"ContainerStarted","Data":"22f76d2e270092f768da9c56c1a65239fa0b1b05fe26aa2b4c84193f3927efd7"}
Jul 21 18:01:19.942140 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:19.942108    2562 generic.go:356] "Generic (PLEG): container finished" podID="0336f600-50f6-4113-a741-a8969761127f" containerID="d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068" exitCode=0
Jul 21 18:01:19.942510 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:19.942177    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz" event={"ID":"0336f600-50f6-4113-a741-a8969761127f","Type":"ContainerDied","Data":"d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068"}
Jul 21 18:01:20.947883 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:20.947851    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz" event={"ID":"0336f600-50f6-4113-a741-a8969761127f","Type":"ContainerStarted","Data":"608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d"}
Jul 21 18:01:20.948385 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:20.947917    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz" event={"ID":"0336f600-50f6-4113-a741-a8969761127f","Type":"ContainerStarted","Data":"5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e"}
Jul 21 18:01:20.948385 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:20.948133    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:20.969598 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:20.969544    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz" podStartSLOduration=3.149917725 podStartE2EDuration="6.969527169s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.819609444s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:01:20.967820386 +0000 UTC m=+2132.539442164" watchObservedRunningTime="2026-07-21 18:01:20.969527169 +0000 UTC m=+2132.541148924"
Jul 21 18:01:21.951857 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:21.951819    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:21.953244 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:21.953216    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz" podUID="0336f600-50f6-4113-a741-a8969761127f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 21 18:01:22.956321 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:22.956281    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz" podUID="0336f600-50f6-4113-a741-a8969761127f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 21 18:01:27.960352 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:27.960326    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:27.961593 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:27.961574    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:34.993396 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:34.993363    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"]
Jul 21 18:01:34.993819 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:34.993587    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz" podUID="0336f600-50f6-4113-a741-a8969761127f" containerName="kserve-container" containerID="cri-o://5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e" gracePeriod=30
Jul 21 18:01:34.993819 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:34.993634    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz" podUID="0336f600-50f6-4113-a741-a8969761127f" containerName="kube-rbac-proxy" containerID="cri-o://608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d" gracePeriod=30
Jul 21 18:01:35.635113 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:35.635089    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:35.689909 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:35.689882    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0336f600-50f6-4113-a741-a8969761127f-proxy-tls\") pod \"0336f600-50f6-4113-a741-a8969761127f\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") "
Jul 21 18:01:35.690030 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:35.689915    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0336f600-50f6-4113-a741-a8969761127f-kserve-provision-location\") pod \"0336f600-50f6-4113-a741-a8969761127f\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") "
Jul 21 18:01:35.690030 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:35.689961    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0336f600-50f6-4113-a741-a8969761127f-cabundle-cert\") pod \"0336f600-50f6-4113-a741-a8969761127f\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") "
Jul 21 18:01:35.690030 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:35.689983    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw7lv\" (UniqueName: \"kubernetes.io/projected/0336f600-50f6-4113-a741-a8969761127f-kube-api-access-jw7lv\") pod \"0336f600-50f6-4113-a741-a8969761127f\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") "
Jul 21 18:01:35.690030 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:35.690020    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0336f600-50f6-4113-a741-a8969761127f-kube-rbac-proxy-sar-config\") pod \"0336f600-50f6-4113-a741-a8969761127f\" (UID: \"0336f600-50f6-4113-a741-a8969761127f\") "
Jul 21 18:01:35.690388 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:35.690359    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0336f600-50f6-4113-a741-a8969761127f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0336f600-50f6-4113-a741-a8969761127f" (UID: "0336f600-50f6-4113-a741-a8969761127f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:01:35.690388 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:35.690359    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0336f600-50f6-4113-a741-a8969761127f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0336f600-50f6-4113-a741-a8969761127f" (UID: "0336f600-50f6-4113-a741-a8969761127f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:01:35.690737 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:35.690425    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0336f600-50f6-4113-a741-a8969761127f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0336f600-50f6-4113-a741-a8969761127f" (UID: "0336f600-50f6-4113-a741-a8969761127f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:01:35.692091 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:35.692052    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0336f600-50f6-4113-a741-a8969761127f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0336f600-50f6-4113-a741-a8969761127f" (UID: "0336f600-50f6-4113-a741-a8969761127f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:01:35.692189 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:35.692126    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0336f600-50f6-4113-a741-a8969761127f-kube-api-access-jw7lv" (OuterVolumeSpecName: "kube-api-access-jw7lv") pod "0336f600-50f6-4113-a741-a8969761127f" (UID: "0336f600-50f6-4113-a741-a8969761127f"). InnerVolumeSpecName "kube-api-access-jw7lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:01:35.790898 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:35.790860    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0336f600-50f6-4113-a741-a8969761127f-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:01:35.790898 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:35.790896    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw7lv\" (UniqueName: \"kubernetes.io/projected/0336f600-50f6-4113-a741-a8969761127f-kube-api-access-jw7lv\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:01:35.790898 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:35.790909    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0336f600-50f6-4113-a741-a8969761127f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:01:35.791140 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:35.790919    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0336f600-50f6-4113-a741-a8969761127f-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:01:35.791140 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:35.790927    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0336f600-50f6-4113-a741-a8969761127f-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:01:36.020589 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.020509    2562 generic.go:356] "Generic (PLEG): container finished" podID="0336f600-50f6-4113-a741-a8969761127f" containerID="608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d" exitCode=2
Jul 21 18:01:36.020589 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.020532    2562 generic.go:356] "Generic (PLEG): container finished" podID="0336f600-50f6-4113-a741-a8969761127f" containerID="5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e" exitCode=0
Jul 21 18:01:36.020589 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.020566    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"
Jul 21 18:01:36.020589 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.020577    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz" event={"ID":"0336f600-50f6-4113-a741-a8969761127f","Type":"ContainerDied","Data":"608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d"}
Jul 21 18:01:36.021226 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.020623    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz" event={"ID":"0336f600-50f6-4113-a741-a8969761127f","Type":"ContainerDied","Data":"5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e"}
Jul 21 18:01:36.021226 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.020642    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz" event={"ID":"0336f600-50f6-4113-a741-a8969761127f","Type":"ContainerDied","Data":"22f76d2e270092f768da9c56c1a65239fa0b1b05fe26aa2b4c84193f3927efd7"}
Jul 21 18:01:36.021226 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.020666    2562 scope.go:117] "RemoveContainer" containerID="608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d"
Jul 21 18:01:36.029303 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.029288    2562 scope.go:117] "RemoveContainer" containerID="5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e"
Jul 21 18:01:36.036414 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.036395    2562 scope.go:117] "RemoveContainer" containerID="d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068"
Jul 21 18:01:36.041529 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.041495    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"]
Jul 21 18:01:36.043875 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.043859    2562 scope.go:117] "RemoveContainer" containerID="608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d"
Jul 21 18:01:36.044128 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:01:36.044110    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d\": container with ID starting with 608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d not found: ID does not exist" containerID="608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d"
Jul 21 18:01:36.044172 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.044136    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d"} err="failed to get container status \"608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d\": rpc error: code = NotFound desc = could not find container \"608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d\": container with ID starting with 608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d not found: ID does not exist"
Jul 21 18:01:36.044172 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.044155    2562 scope.go:117] "RemoveContainer" containerID="5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e"
Jul 21 18:01:36.044409 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:01:36.044392    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e\": container with ID starting with 5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e not found: ID does not exist" containerID="5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e"
Jul 21 18:01:36.044595 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.044416    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e"} err="failed to get container status \"5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e\": rpc error: code = NotFound desc = could not find container \"5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e\": container with ID starting with 5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e not found: ID does not exist"
Jul 21 18:01:36.044595 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.044434    2562 scope.go:117] "RemoveContainer" containerID="d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068"
Jul 21 18:01:36.044724 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:01:36.044678    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068\": container with ID starting with d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068 not found: ID does not exist" containerID="d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068"
Jul 21 18:01:36.044724 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.044700    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068"} err="failed to get container status \"d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068\": rpc error: code = NotFound desc = could not find container \"d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068\": container with ID starting with d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068 not found: ID does not exist"
Jul 21 18:01:36.044724 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.044720    2562 scope.go:117] "RemoveContainer" containerID="608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d"
Jul 21 18:01:36.045036 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.045009    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d"} err="failed to get container status \"608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d\": rpc error: code = NotFound desc = could not find container \"608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d\": container with ID starting with 608ee9253a1a0bf7961eb251d192aa46c4fc16e547a6998394be1b97f832366d not found: ID does not exist"
Jul 21 18:01:36.045036 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.045038    2562 scope.go:117] "RemoveContainer" containerID="5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e"
Jul 21 18:01:36.045382 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.045358    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e"} err="failed to get container status \"5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e\": rpc error: code = NotFound desc = could not find container \"5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e\": container with ID starting with 5a82a91edd80a9992197373f71c0e8116318834f3e5fc95adf73b7c16545521e not found: ID does not exist"
Jul 21 18:01:36.045458 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.045385    2562 scope.go:117] "RemoveContainer" containerID="d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068"
Jul 21 18:01:36.045683 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.045661    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068"} err="failed to get container status \"d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068\": rpc error: code = NotFound desc = could not find container \"d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068\": container with ID starting with d30058ae672798d1837e737377e137c743d7c798ae10e694d498d39630363068 not found: ID does not exist"
Jul 21 18:01:36.047026 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.047008    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-7lwmz"]
Jul 21 18:01:36.921405 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:36.921365    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0336f600-50f6-4113-a741-a8969761127f" path="/var/lib/kubelet/pods/0336f600-50f6-4113-a741-a8969761127f/volumes"
Jul 21 18:01:40.174293 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.174210    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"]
Jul 21 18:01:40.174644 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.174608    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0336f600-50f6-4113-a741-a8969761127f" containerName="kserve-container"
Jul 21 18:01:40.174644 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.174621    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0336f600-50f6-4113-a741-a8969761127f" containerName="kserve-container"
Jul 21 18:01:40.174644 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.174634    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0336f600-50f6-4113-a741-a8969761127f" containerName="storage-initializer"
Jul 21 18:01:40.174644 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.174640    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0336f600-50f6-4113-a741-a8969761127f" containerName="storage-initializer"
Jul 21 18:01:40.174644 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.174647    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0336f600-50f6-4113-a741-a8969761127f" containerName="kube-rbac-proxy"
Jul 21 18:01:40.174818 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.174652    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0336f600-50f6-4113-a741-a8969761127f" containerName="kube-rbac-proxy"
Jul 21 18:01:40.174818 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.174708    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0336f600-50f6-4113-a741-a8969761127f" containerName="kserve-container"
Jul 21 18:01:40.174818 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.174718    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0336f600-50f6-4113-a741-a8969761127f" containerName="kube-rbac-proxy"
Jul 21 18:01:40.176784 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.176766    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:40.179617 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.179593    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 21 18:01:40.179617 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.179611    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:01:40.179816 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.179611    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:01:40.179816 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.179611    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:01:40.179816 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.179616    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:01:40.180611 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.180592    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 21 18:01:40.180717 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.180624    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:01:40.188140 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.188119    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"]
Jul 21 18:01:40.224840 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.224812    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77b884d9-81f1-4726-b3ea-84fa286d036e-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:40.224973 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.224860    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27zvn\" (UniqueName: \"kubernetes.io/projected/77b884d9-81f1-4726-b3ea-84fa286d036e-kube-api-access-27zvn\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:40.224973 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.224881    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77b884d9-81f1-4726-b3ea-84fa286d036e-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:40.224973 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.224951    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77b884d9-81f1-4726-b3ea-84fa286d036e-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:40.225119 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.224986    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77b884d9-81f1-4726-b3ea-84fa286d036e-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:40.326083 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.326033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77b884d9-81f1-4726-b3ea-84fa286d036e-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:40.326216 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.326091    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77b884d9-81f1-4726-b3ea-84fa286d036e-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:40.326216 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.326127    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77b884d9-81f1-4726-b3ea-84fa286d036e-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:40.326216 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.326168    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27zvn\" (UniqueName: \"kubernetes.io/projected/77b884d9-81f1-4726-b3ea-84fa286d036e-kube-api-access-27zvn\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:40.326216 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:01:40.326180    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 21 18:01:40.326216 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.326191    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77b884d9-81f1-4726-b3ea-84fa286d036e-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:40.326506 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:01:40.326235    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77b884d9-81f1-4726-b3ea-84fa286d036e-proxy-tls podName:77b884d9-81f1-4726-b3ea-84fa286d036e nodeName:}" failed. No retries permitted until 2026-07-21 18:01:40.826217852 +0000 UTC m=+2152.397839585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/77b884d9-81f1-4726-b3ea-84fa286d036e-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5" (UID: "77b884d9-81f1-4726-b3ea-84fa286d036e") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 21 18:01:40.326639 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.326619    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77b884d9-81f1-4726-b3ea-84fa286d036e-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:40.326890 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.326874    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77b884d9-81f1-4726-b3ea-84fa286d036e-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:40.326951 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.326935    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77b884d9-81f1-4726-b3ea-84fa286d036e-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:40.334958 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.334936    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27zvn\" (UniqueName: \"kubernetes.io/projected/77b884d9-81f1-4726-b3ea-84fa286d036e-kube-api-access-27zvn\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:40.830336 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.830303    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77b884d9-81f1-4726-b3ea-84fa286d036e-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:40.832759 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:40.832727    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77b884d9-81f1-4726-b3ea-84fa286d036e-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:41.087632 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:41.087552    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:41.223169 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:41.223133    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"]
Jul 21 18:01:41.225613 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:01:41.225580    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b884d9_81f1_4726_b3ea_84fa286d036e.slice/crio-e4ab3afad204ac813bd57302649dc59c1466ab566611fa967260bb023fc0a2d8 WatchSource:0}: Error finding container e4ab3afad204ac813bd57302649dc59c1466ab566611fa967260bb023fc0a2d8: Status 404 returned error can't find the container with id e4ab3afad204ac813bd57302649dc59c1466ab566611fa967260bb023fc0a2d8
Jul 21 18:01:42.047660 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:42.047624    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5" event={"ID":"77b884d9-81f1-4726-b3ea-84fa286d036e","Type":"ContainerStarted","Data":"c49845ffe823daff09af7fde3dcebbaf49ef20178feae3217a265f3689c4df25"}
Jul 21 18:01:42.047843 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:42.047681    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5" event={"ID":"77b884d9-81f1-4726-b3ea-84fa286d036e","Type":"ContainerStarted","Data":"e4ab3afad204ac813bd57302649dc59c1466ab566611fa967260bb023fc0a2d8"}
Jul 21 18:01:46.066162 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:46.066126    2562 generic.go:356] "Generic (PLEG): container finished" podID="77b884d9-81f1-4726-b3ea-84fa286d036e" containerID="c49845ffe823daff09af7fde3dcebbaf49ef20178feae3217a265f3689c4df25" exitCode=0
Jul 21 18:01:46.066565 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:46.066200    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5" event={"ID":"77b884d9-81f1-4726-b3ea-84fa286d036e","Type":"ContainerDied","Data":"c49845ffe823daff09af7fde3dcebbaf49ef20178feae3217a265f3689c4df25"}
Jul 21 18:01:47.072911 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:47.072867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5" event={"ID":"77b884d9-81f1-4726-b3ea-84fa286d036e","Type":"ContainerStarted","Data":"ec06395e0d73ff88ab11031df580733474f5b0c1281f2d302e76401adcaab582"}
Jul 21 18:01:47.073313 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:47.072950    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5" event={"ID":"77b884d9-81f1-4726-b3ea-84fa286d036e","Type":"ContainerStarted","Data":"0dd2e712ebe09dc64ffc46f47f20c6462be0ad5512d07af97b9da8da06c5d742"}
Jul 21 18:01:47.073313 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:47.073200    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:47.098723 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:47.098676    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5" podStartSLOduration=2.988264603 podStartE2EDuration="7.098661372s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.110396769s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:01:47.097215691 +0000 UTC m=+2158.668837509" watchObservedRunningTime="2026-07-21 18:01:47.098661372 +0000 UTC m=+2158.670283126"
Jul 21 18:01:48.077313 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:48.077284    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:01:54.086548 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:01:54.086513    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:02:24.091112 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:24.091082    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:02:30.277408 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:30.277373    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"]
Jul 21 18:02:30.277925 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:30.277608    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5" podUID="77b884d9-81f1-4726-b3ea-84fa286d036e" containerName="kserve-container" containerID="cri-o://0dd2e712ebe09dc64ffc46f47f20c6462be0ad5512d07af97b9da8da06c5d742" gracePeriod=30
Jul 21 18:02:30.277925 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:30.277630    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5" podUID="77b884d9-81f1-4726-b3ea-84fa286d036e" containerName="kube-rbac-proxy" containerID="cri-o://ec06395e0d73ff88ab11031df580733474f5b0c1281f2d302e76401adcaab582" gracePeriod=30
Jul 21 18:02:31.262267 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.262229    2562 generic.go:356] "Generic (PLEG): container finished" podID="77b884d9-81f1-4726-b3ea-84fa286d036e" containerID="ec06395e0d73ff88ab11031df580733474f5b0c1281f2d302e76401adcaab582" exitCode=2
Jul 21 18:02:31.262457 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.262289    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5" event={"ID":"77b884d9-81f1-4726-b3ea-84fa286d036e","Type":"ContainerDied","Data":"ec06395e0d73ff88ab11031df580733474f5b0c1281f2d302e76401adcaab582"}
Jul 21 18:02:31.538144 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.538112    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:02:31.556039 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.556012    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77b884d9-81f1-4726-b3ea-84fa286d036e-cabundle-cert\") pod \"77b884d9-81f1-4726-b3ea-84fa286d036e\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") "
Jul 21 18:02:31.556200 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.556049    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27zvn\" (UniqueName: \"kubernetes.io/projected/77b884d9-81f1-4726-b3ea-84fa286d036e-kube-api-access-27zvn\") pod \"77b884d9-81f1-4726-b3ea-84fa286d036e\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") "
Jul 21 18:02:31.556200 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.556138    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77b884d9-81f1-4726-b3ea-84fa286d036e-proxy-tls\") pod \"77b884d9-81f1-4726-b3ea-84fa286d036e\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") "
Jul 21 18:02:31.556275 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.556203    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77b884d9-81f1-4726-b3ea-84fa286d036e-kserve-provision-location\") pod \"77b884d9-81f1-4726-b3ea-84fa286d036e\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") "
Jul 21 18:02:31.556412 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.556387    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b884d9-81f1-4726-b3ea-84fa286d036e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "77b884d9-81f1-4726-b3ea-84fa286d036e" (UID: "77b884d9-81f1-4726-b3ea-84fa286d036e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:02:31.556582 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.556554    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b884d9-81f1-4726-b3ea-84fa286d036e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "77b884d9-81f1-4726-b3ea-84fa286d036e" (UID: "77b884d9-81f1-4726-b3ea-84fa286d036e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:02:31.558329 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.558308    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77b884d9-81f1-4726-b3ea-84fa286d036e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77b884d9-81f1-4726-b3ea-84fa286d036e" (UID: "77b884d9-81f1-4726-b3ea-84fa286d036e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:02:31.558437 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.558395    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b884d9-81f1-4726-b3ea-84fa286d036e-kube-api-access-27zvn" (OuterVolumeSpecName: "kube-api-access-27zvn") pod "77b884d9-81f1-4726-b3ea-84fa286d036e" (UID: "77b884d9-81f1-4726-b3ea-84fa286d036e"). InnerVolumeSpecName "kube-api-access-27zvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:02:31.656851 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.656816    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77b884d9-81f1-4726-b3ea-84fa286d036e-kube-rbac-proxy-sar-config\") pod \"77b884d9-81f1-4726-b3ea-84fa286d036e\" (UID: \"77b884d9-81f1-4726-b3ea-84fa286d036e\") "
Jul 21 18:02:31.657036 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.657019    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27zvn\" (UniqueName: \"kubernetes.io/projected/77b884d9-81f1-4726-b3ea-84fa286d036e-kube-api-access-27zvn\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:02:31.657124 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.657044    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77b884d9-81f1-4726-b3ea-84fa286d036e-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:02:31.657124 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.657058    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77b884d9-81f1-4726-b3ea-84fa286d036e-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:02:31.657124 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.657112    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77b884d9-81f1-4726-b3ea-84fa286d036e-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:02:31.657259 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.657243    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b884d9-81f1-4726-b3ea-84fa286d036e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "77b884d9-81f1-4726-b3ea-84fa286d036e" (UID: "77b884d9-81f1-4726-b3ea-84fa286d036e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:02:31.757634 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:31.757600    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77b884d9-81f1-4726-b3ea-84fa286d036e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:02:32.268224 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:32.268190    2562 generic.go:356] "Generic (PLEG): container finished" podID="77b884d9-81f1-4726-b3ea-84fa286d036e" containerID="0dd2e712ebe09dc64ffc46f47f20c6462be0ad5512d07af97b9da8da06c5d742" exitCode=0
Jul 21 18:02:32.268396 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:32.268252    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"
Jul 21 18:02:32.268396 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:32.268279    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5" event={"ID":"77b884d9-81f1-4726-b3ea-84fa286d036e","Type":"ContainerDied","Data":"0dd2e712ebe09dc64ffc46f47f20c6462be0ad5512d07af97b9da8da06c5d742"}
Jul 21 18:02:32.268396 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:32.268349    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5" event={"ID":"77b884d9-81f1-4726-b3ea-84fa286d036e","Type":"ContainerDied","Data":"e4ab3afad204ac813bd57302649dc59c1466ab566611fa967260bb023fc0a2d8"}
Jul 21 18:02:32.268396 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:32.268387    2562 scope.go:117] "RemoveContainer" containerID="ec06395e0d73ff88ab11031df580733474f5b0c1281f2d302e76401adcaab582"
Jul 21 18:02:32.279573 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:32.279553    2562 scope.go:117] "RemoveContainer" containerID="0dd2e712ebe09dc64ffc46f47f20c6462be0ad5512d07af97b9da8da06c5d742"
Jul 21 18:02:32.287950 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:32.287926    2562 scope.go:117] "RemoveContainer" containerID="c49845ffe823daff09af7fde3dcebbaf49ef20178feae3217a265f3689c4df25"
Jul 21 18:02:32.293200 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:32.293178    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"]
Jul 21 18:02:32.295762 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:32.295742    2562 scope.go:117] "RemoveContainer" containerID="ec06395e0d73ff88ab11031df580733474f5b0c1281f2d302e76401adcaab582"
Jul 21 18:02:32.296024 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:02:32.296005    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec06395e0d73ff88ab11031df580733474f5b0c1281f2d302e76401adcaab582\": container with ID starting with ec06395e0d73ff88ab11031df580733474f5b0c1281f2d302e76401adcaab582 not found: ID does not exist" containerID="ec06395e0d73ff88ab11031df580733474f5b0c1281f2d302e76401adcaab582"
Jul 21 18:02:32.296120 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:32.296037    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec06395e0d73ff88ab11031df580733474f5b0c1281f2d302e76401adcaab582"} err="failed to get container status \"ec06395e0d73ff88ab11031df580733474f5b0c1281f2d302e76401adcaab582\": rpc error: code = NotFound desc = could not find container \"ec06395e0d73ff88ab11031df580733474f5b0c1281f2d302e76401adcaab582\": container with ID starting with ec06395e0d73ff88ab11031df580733474f5b0c1281f2d302e76401adcaab582 not found: ID does not exist"
Jul 21 18:02:32.296120 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:32.296058    2562 scope.go:117] "RemoveContainer" containerID="0dd2e712ebe09dc64ffc46f47f20c6462be0ad5512d07af97b9da8da06c5d742"
Jul 21 18:02:32.296345 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:02:32.296326    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd2e712ebe09dc64ffc46f47f20c6462be0ad5512d07af97b9da8da06c5d742\": container with ID starting with 0dd2e712ebe09dc64ffc46f47f20c6462be0ad5512d07af97b9da8da06c5d742 not found: ID does not exist" containerID="0dd2e712ebe09dc64ffc46f47f20c6462be0ad5512d07af97b9da8da06c5d742"
Jul 21 18:02:32.296386 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:32.296351    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd2e712ebe09dc64ffc46f47f20c6462be0ad5512d07af97b9da8da06c5d742"} err="failed to get container status \"0dd2e712ebe09dc64ffc46f47f20c6462be0ad5512d07af97b9da8da06c5d742\": rpc error: code = NotFound desc = could not find container \"0dd2e712ebe09dc64ffc46f47f20c6462be0ad5512d07af97b9da8da06c5d742\": container with ID starting with 0dd2e712ebe09dc64ffc46f47f20c6462be0ad5512d07af97b9da8da06c5d742 not found: ID does not exist"
Jul 21 18:02:32.296386 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:32.296366    2562 scope.go:117] "RemoveContainer" containerID="c49845ffe823daff09af7fde3dcebbaf49ef20178feae3217a265f3689c4df25"
Jul 21 18:02:32.296601 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:02:32.296584    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49845ffe823daff09af7fde3dcebbaf49ef20178feae3217a265f3689c4df25\": container with ID starting with c49845ffe823daff09af7fde3dcebbaf49ef20178feae3217a265f3689c4df25 not found: ID does not exist" containerID="c49845ffe823daff09af7fde3dcebbaf49ef20178feae3217a265f3689c4df25"
Jul 21 18:02:32.296649 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:32.296606    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49845ffe823daff09af7fde3dcebbaf49ef20178feae3217a265f3689c4df25"} err="failed to get container status \"c49845ffe823daff09af7fde3dcebbaf49ef20178feae3217a265f3689c4df25\": rpc error: code = NotFound desc = could not find container \"c49845ffe823daff09af7fde3dcebbaf49ef20178feae3217a265f3689c4df25\": container with ID starting with c49845ffe823daff09af7fde3dcebbaf49ef20178feae3217a265f3689c4df25 not found: ID does not exist"
Jul 21 18:02:32.299218 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:32.299196    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xxhh5"]
Jul 21 18:02:32.921632 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:32.921598    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77b884d9-81f1-4726-b3ea-84fa286d036e" path="/var/lib/kubelet/pods/77b884d9-81f1-4726-b3ea-84fa286d036e/volumes"
Jul 21 18:02:35.437317 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.437285    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"]
Jul 21 18:02:35.437710 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.437617    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77b884d9-81f1-4726-b3ea-84fa286d036e" containerName="storage-initializer"
Jul 21 18:02:35.437710 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.437629    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="77b884d9-81f1-4726-b3ea-84fa286d036e" containerName="storage-initializer"
Jul 21 18:02:35.437710 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.437642    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77b884d9-81f1-4726-b3ea-84fa286d036e" containerName="kube-rbac-proxy"
Jul 21 18:02:35.437710 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.437651    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="77b884d9-81f1-4726-b3ea-84fa286d036e" containerName="kube-rbac-proxy"
Jul 21 18:02:35.437710 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.437664    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77b884d9-81f1-4726-b3ea-84fa286d036e" containerName="kserve-container"
Jul 21 18:02:35.437710 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.437670    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="77b884d9-81f1-4726-b3ea-84fa286d036e" containerName="kserve-container"
Jul 21 18:02:35.437903 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.437731    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="77b884d9-81f1-4726-b3ea-84fa286d036e" containerName="kube-rbac-proxy"
Jul 21 18:02:35.437903 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.437743    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="77b884d9-81f1-4726-b3ea-84fa286d036e" containerName="kserve-container"
Jul 21 18:02:35.441299 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.441280    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:35.443805 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.443786    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 21 18:02:35.444051 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.444034    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:02:35.444161 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.444053    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 21 18:02:35.444161 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.444084    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:02:35.444161 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.444105    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:02:35.444351 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.444334    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:02:35.444414 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.444391    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:02:35.452453 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.452434    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"]
Jul 21 18:02:35.484696 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.484666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvbcj\" (UniqueName: \"kubernetes.io/projected/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kube-api-access-wvbcj\") pod \"isvc-sklearn-mcp-predictor-5f988fb756-sn9cf\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:35.484874 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.484709    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5f988fb756-sn9cf\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:35.484874 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.484735    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5f988fb756-sn9cf\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:35.484874 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.484817    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5f988fb756-sn9cf\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:35.484874 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.484867    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5f988fb756-sn9cf\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:35.585259 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.585219    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvbcj\" (UniqueName: \"kubernetes.io/projected/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kube-api-access-wvbcj\") pod \"isvc-sklearn-mcp-predictor-5f988fb756-sn9cf\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:35.585416 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.585272    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5f988fb756-sn9cf\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:35.585416 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.585322    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5f988fb756-sn9cf\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:35.585416 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:02:35.585369    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Jul 21 18:02:35.585416 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.585385    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5f988fb756-sn9cf\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:35.585630 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:02:35.585434    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-proxy-tls podName:e6b0cd43-b479-49e0-b918-f9ad25c44cd5 nodeName:}" failed. No retries permitted until 2026-07-21 18:02:36.085410958 +0000 UTC m=+2207.657032692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-proxy-tls") pod "isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" (UID: "e6b0cd43-b479-49e0-b918-f9ad25c44cd5") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Jul 21 18:02:35.585630 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.585480    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5f988fb756-sn9cf\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:35.585939 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.585919    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5f988fb756-sn9cf\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:35.586041 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.586025    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5f988fb756-sn9cf\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:35.586174 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.586158    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5f988fb756-sn9cf\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:35.594324 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:35.594297    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvbcj\" (UniqueName: \"kubernetes.io/projected/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kube-api-access-wvbcj\") pod \"isvc-sklearn-mcp-predictor-5f988fb756-sn9cf\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:36.089078 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:36.089023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5f988fb756-sn9cf\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:36.091599 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:36.091565    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5f988fb756-sn9cf\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:36.352110 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:36.351985    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:36.491819 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:36.491775    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"]
Jul 21 18:02:36.498401 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:02:36.498367    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b0cd43_b479_49e0_b918_f9ad25c44cd5.slice/crio-a4381017035315c2170928a373529eb8a4b3ba8628deca1c9f3d945ffd4ff063 WatchSource:0}: Error finding container a4381017035315c2170928a373529eb8a4b3ba8628deca1c9f3d945ffd4ff063: Status 404 returned error can't find the container with id a4381017035315c2170928a373529eb8a4b3ba8628deca1c9f3d945ffd4ff063
Jul 21 18:02:37.291274 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:37.291241    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" event={"ID":"e6b0cd43-b479-49e0-b918-f9ad25c44cd5","Type":"ContainerStarted","Data":"a81e61e5cdcb82ac9e41e06bc74d2c6240ac3bc4297ff6ab8c0f1fe7a0d7dcfe"}
Jul 21 18:02:37.291464 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:37.291304    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" event={"ID":"e6b0cd43-b479-49e0-b918-f9ad25c44cd5","Type":"ContainerStarted","Data":"a4381017035315c2170928a373529eb8a4b3ba8628deca1c9f3d945ffd4ff063"}
Jul 21 18:02:40.305792 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:40.305758    2562 generic.go:356] "Generic (PLEG): container finished" podID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerID="a81e61e5cdcb82ac9e41e06bc74d2c6240ac3bc4297ff6ab8c0f1fe7a0d7dcfe" exitCode=0
Jul 21 18:02:40.306217 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:40.305806    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" event={"ID":"e6b0cd43-b479-49e0-b918-f9ad25c44cd5","Type":"ContainerDied","Data":"a81e61e5cdcb82ac9e41e06bc74d2c6240ac3bc4297ff6ab8c0f1fe7a0d7dcfe"}
Jul 21 18:02:40.363349 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:40.363326    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 18:02:41.313479 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:41.313442    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" event={"ID":"e6b0cd43-b479-49e0-b918-f9ad25c44cd5","Type":"ContainerStarted","Data":"57f6841f12502582ea0577a3210e6afead4c3bdf82cb291b365d9f6987afbef4"}
Jul 21 18:02:43.330769 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:43.330680    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" event={"ID":"e6b0cd43-b479-49e0-b918-f9ad25c44cd5","Type":"ContainerStarted","Data":"fb2c14192ef8a51ca8c522ecfeaf626dc7e82ff846a063ab42df764f93ae5cb1"}
Jul 21 18:02:43.330769 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:43.330753    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" event={"ID":"e6b0cd43-b479-49e0-b918-f9ad25c44cd5","Type":"ContainerStarted","Data":"1e6821e7d1ff273eebb563caf344e5ee764158a9f4850befdd1c8db99e5d34a5"}
Jul 21 18:02:43.331207 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:43.330908    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:43.331207 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:43.330981    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:43.357899 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:43.357847    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" podStartSLOduration=2.078349584 podStartE2EDuration="8.357818819s" totalImagesPullingTime="2.669282407s" totalInitContainerRuntime="3.610186828s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:02:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:02:43.355603945 +0000 UTC m=+2214.927225721" watchObservedRunningTime="2026-07-21 18:02:43.357818819 +0000 UTC m=+2214.929440573"
Jul 21 18:02:44.335013 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:44.334981    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:02:50.346422 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:02:50.346384    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:03:10.347434 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:03:10.347339    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 21 18:03:20.348572 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:03:20.348540    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:03:50.349432 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:03:50.349395    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:03:55.583288 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:03:55.583255    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"]
Jul 21 18:03:55.583717 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:03:55.583567    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kube-rbac-proxy" containerID="cri-o://fb2c14192ef8a51ca8c522ecfeaf626dc7e82ff846a063ab42df764f93ae5cb1" gracePeriod=30
Jul 21 18:03:55.583717 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:03:55.583597    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kserve-agent" containerID="cri-o://1e6821e7d1ff273eebb563caf344e5ee764158a9f4850befdd1c8db99e5d34a5" gracePeriod=30
Jul 21 18:03:55.583717 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:03:55.583553    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kserve-container" containerID="cri-o://57f6841f12502582ea0577a3210e6afead4c3bdf82cb291b365d9f6987afbef4" gracePeriod=30
Jul 21 18:03:56.629570 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:03:56.629537    2562 generic.go:356] "Generic (PLEG): container finished" podID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerID="fb2c14192ef8a51ca8c522ecfeaf626dc7e82ff846a063ab42df764f93ae5cb1" exitCode=2
Jul 21 18:03:56.629947 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:03:56.629610    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" event={"ID":"e6b0cd43-b479-49e0-b918-f9ad25c44cd5","Type":"ContainerDied","Data":"fb2c14192ef8a51ca8c522ecfeaf626dc7e82ff846a063ab42df764f93ae5cb1"}
Jul 21 18:03:57.636173 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:03:57.636146    2562 generic.go:356] "Generic (PLEG): container finished" podID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerID="57f6841f12502582ea0577a3210e6afead4c3bdf82cb291b365d9f6987afbef4" exitCode=0
Jul 21 18:03:57.636491 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:03:57.636226    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" event={"ID":"e6b0cd43-b479-49e0-b918-f9ad25c44cd5","Type":"ContainerDied","Data":"57f6841f12502582ea0577a3210e6afead4c3bdf82cb291b365d9f6987afbef4"}
Jul 21 18:04:00.340489 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:00.340441    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 21 18:04:00.346997 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:00.346961    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 21 18:04:05.339885 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:05.339833    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 21 18:04:10.339833 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:10.339793    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 21 18:04:10.340218 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:10.339913    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:04:10.347269 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:10.347239    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 21 18:04:15.339675 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:15.339637    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 21 18:04:20.340397 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:20.340357    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 21 18:04:20.346927 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:20.346894    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 21 18:04:20.347109 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:20.347092    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:04:25.339833 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.339797    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 21 18:04:25.732816 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.732789    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:04:25.746284 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.746260    2562 generic.go:356] "Generic (PLEG): container finished" podID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerID="1e6821e7d1ff273eebb563caf344e5ee764158a9f4850befdd1c8db99e5d34a5" exitCode=0
Jul 21 18:04:25.746389 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.746309    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"
Jul 21 18:04:25.746389 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.746333    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" event={"ID":"e6b0cd43-b479-49e0-b918-f9ad25c44cd5","Type":"ContainerDied","Data":"1e6821e7d1ff273eebb563caf344e5ee764158a9f4850befdd1c8db99e5d34a5"}
Jul 21 18:04:25.746465 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.746407    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf" event={"ID":"e6b0cd43-b479-49e0-b918-f9ad25c44cd5","Type":"ContainerDied","Data":"a4381017035315c2170928a373529eb8a4b3ba8628deca1c9f3d945ffd4ff063"}
Jul 21 18:04:25.746465 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.746451    2562 scope.go:117] "RemoveContainer" containerID="fb2c14192ef8a51ca8c522ecfeaf626dc7e82ff846a063ab42df764f93ae5cb1"
Jul 21 18:04:25.754439 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.754424    2562 scope.go:117] "RemoveContainer" containerID="1e6821e7d1ff273eebb563caf344e5ee764158a9f4850befdd1c8db99e5d34a5"
Jul 21 18:04:25.768079 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.768043    2562 scope.go:117] "RemoveContainer" containerID="57f6841f12502582ea0577a3210e6afead4c3bdf82cb291b365d9f6987afbef4"
Jul 21 18:04:25.775240 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.775216    2562 scope.go:117] "RemoveContainer" containerID="a81e61e5cdcb82ac9e41e06bc74d2c6240ac3bc4297ff6ab8c0f1fe7a0d7dcfe"
Jul 21 18:04:25.783792 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.783776    2562 scope.go:117] "RemoveContainer" containerID="fb2c14192ef8a51ca8c522ecfeaf626dc7e82ff846a063ab42df764f93ae5cb1"
Jul 21 18:04:25.784077 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:04:25.784037    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb2c14192ef8a51ca8c522ecfeaf626dc7e82ff846a063ab42df764f93ae5cb1\": container with ID starting with fb2c14192ef8a51ca8c522ecfeaf626dc7e82ff846a063ab42df764f93ae5cb1 not found: ID does not exist" containerID="fb2c14192ef8a51ca8c522ecfeaf626dc7e82ff846a063ab42df764f93ae5cb1"
Jul 21 18:04:25.784164 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.784092    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2c14192ef8a51ca8c522ecfeaf626dc7e82ff846a063ab42df764f93ae5cb1"} err="failed to get container status \"fb2c14192ef8a51ca8c522ecfeaf626dc7e82ff846a063ab42df764f93ae5cb1\": rpc error: code = NotFound desc = could not find container \"fb2c14192ef8a51ca8c522ecfeaf626dc7e82ff846a063ab42df764f93ae5cb1\": container with ID starting with fb2c14192ef8a51ca8c522ecfeaf626dc7e82ff846a063ab42df764f93ae5cb1 not found: ID does not exist"
Jul 21 18:04:25.784164 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.784118    2562 scope.go:117] "RemoveContainer" containerID="1e6821e7d1ff273eebb563caf344e5ee764158a9f4850befdd1c8db99e5d34a5"
Jul 21 18:04:25.784400 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:04:25.784378    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6821e7d1ff273eebb563caf344e5ee764158a9f4850befdd1c8db99e5d34a5\": container with ID starting with 1e6821e7d1ff273eebb563caf344e5ee764158a9f4850befdd1c8db99e5d34a5 not found: ID does not exist" containerID="1e6821e7d1ff273eebb563caf344e5ee764158a9f4850befdd1c8db99e5d34a5"
Jul 21 18:04:25.784464 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.784413    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6821e7d1ff273eebb563caf344e5ee764158a9f4850befdd1c8db99e5d34a5"} err="failed to get container status \"1e6821e7d1ff273eebb563caf344e5ee764158a9f4850befdd1c8db99e5d34a5\": rpc error: code = NotFound desc = could not find container \"1e6821e7d1ff273eebb563caf344e5ee764158a9f4850befdd1c8db99e5d34a5\": container with ID starting with 1e6821e7d1ff273eebb563caf344e5ee764158a9f4850befdd1c8db99e5d34a5 not found: ID does not exist"
Jul 21 18:04:25.784464 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.784435    2562 scope.go:117] "RemoveContainer" containerID="57f6841f12502582ea0577a3210e6afead4c3bdf82cb291b365d9f6987afbef4"
Jul 21 18:04:25.784730 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:04:25.784708    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f6841f12502582ea0577a3210e6afead4c3bdf82cb291b365d9f6987afbef4\": container with ID starting with 57f6841f12502582ea0577a3210e6afead4c3bdf82cb291b365d9f6987afbef4 not found: ID does not exist" containerID="57f6841f12502582ea0577a3210e6afead4c3bdf82cb291b365d9f6987afbef4"
Jul 21 18:04:25.784781 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.784741    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f6841f12502582ea0577a3210e6afead4c3bdf82cb291b365d9f6987afbef4"} err="failed to get container status \"57f6841f12502582ea0577a3210e6afead4c3bdf82cb291b365d9f6987afbef4\": rpc error: code = NotFound desc = could not find container \"57f6841f12502582ea0577a3210e6afead4c3bdf82cb291b365d9f6987afbef4\": container with ID starting with 57f6841f12502582ea0577a3210e6afead4c3bdf82cb291b365d9f6987afbef4 not found: ID does not exist"
Jul 21 18:04:25.784781 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.784761    2562 scope.go:117] "RemoveContainer" containerID="a81e61e5cdcb82ac9e41e06bc74d2c6240ac3bc4297ff6ab8c0f1fe7a0d7dcfe"
Jul 21 18:04:25.785053 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:04:25.785018    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a81e61e5cdcb82ac9e41e06bc74d2c6240ac3bc4297ff6ab8c0f1fe7a0d7dcfe\": container with ID starting with a81e61e5cdcb82ac9e41e06bc74d2c6240ac3bc4297ff6ab8c0f1fe7a0d7dcfe not found: ID does not exist" containerID="a81e61e5cdcb82ac9e41e06bc74d2c6240ac3bc4297ff6ab8c0f1fe7a0d7dcfe"
Jul 21 18:04:25.785143 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.785079    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a81e61e5cdcb82ac9e41e06bc74d2c6240ac3bc4297ff6ab8c0f1fe7a0d7dcfe"} err="failed to get container status \"a81e61e5cdcb82ac9e41e06bc74d2c6240ac3bc4297ff6ab8c0f1fe7a0d7dcfe\": rpc error: code = NotFound desc = could not find container \"a81e61e5cdcb82ac9e41e06bc74d2c6240ac3bc4297ff6ab8c0f1fe7a0d7dcfe\": container with ID starting with a81e61e5cdcb82ac9e41e06bc74d2c6240ac3bc4297ff6ab8c0f1fe7a0d7dcfe not found: ID does not exist"
Jul 21 18:04:25.890027 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.889973    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kserve-provision-location\") pod \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") "
Jul 21 18:04:25.890027 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.890024    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvbcj\" (UniqueName: \"kubernetes.io/projected/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kube-api-access-wvbcj\") pod \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") "
Jul 21 18:04:25.890206 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.890041    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kube-rbac-proxy-sar-config\") pod \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") "
Jul 21 18:04:25.890206 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.890092    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-proxy-tls\") pod \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") "
Jul 21 18:04:25.890206 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.890124    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-cabundle-cert\") pod \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\" (UID: \"e6b0cd43-b479-49e0-b918-f9ad25c44cd5\") "
Jul 21 18:04:25.890361 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.890312    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6b0cd43-b479-49e0-b918-f9ad25c44cd5" (UID: "e6b0cd43-b479-49e0-b918-f9ad25c44cd5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:04:25.890468 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.890447    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e6b0cd43-b479-49e0-b918-f9ad25c44cd5" (UID: "e6b0cd43-b479-49e0-b918-f9ad25c44cd5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:04:25.890556 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.890530    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e6b0cd43-b479-49e0-b918-f9ad25c44cd5" (UID: "e6b0cd43-b479-49e0-b918-f9ad25c44cd5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:04:25.892027 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.892002    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kube-api-access-wvbcj" (OuterVolumeSpecName: "kube-api-access-wvbcj") pod "e6b0cd43-b479-49e0-b918-f9ad25c44cd5" (UID: "e6b0cd43-b479-49e0-b918-f9ad25c44cd5"). InnerVolumeSpecName "kube-api-access-wvbcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:04:25.892124 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.892085    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e6b0cd43-b479-49e0-b918-f9ad25c44cd5" (UID: "e6b0cd43-b479-49e0-b918-f9ad25c44cd5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:04:25.991506 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.991469    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:04:25.991506 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.991504    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:04:25.991642 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.991520    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:04:25.991642 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.991532    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvbcj\" (UniqueName: \"kubernetes.io/projected/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kube-api-access-wvbcj\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:04:25.991642 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:25.991544    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6b0cd43-b479-49e0-b918-f9ad25c44cd5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:04:26.070183 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:26.070158    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"]
Jul 21 18:04:26.076623 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:26.076599    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5f988fb756-sn9cf"]
Jul 21 18:04:26.920965 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:04:26.920926    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" path="/var/lib/kubelet/pods/e6b0cd43-b479-49e0-b918-f9ad25c44cd5/volumes"
Jul 21 18:05:17.586994 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.586956    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"]
Jul 21 18:05:17.587460 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.587310    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kserve-agent"
Jul 21 18:05:17.587460 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.587321    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kserve-agent"
Jul 21 18:05:17.587460 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.587334    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="storage-initializer"
Jul 21 18:05:17.587460 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.587340    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="storage-initializer"
Jul 21 18:05:17.587460 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.587353    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kube-rbac-proxy"
Jul 21 18:05:17.587460 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.587359    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kube-rbac-proxy"
Jul 21 18:05:17.587460 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.587367    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kserve-container"
Jul 21 18:05:17.587460 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.587372    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kserve-container"
Jul 21 18:05:17.587460 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.587426    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kserve-agent"
Jul 21 18:05:17.587460 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.587433    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kserve-container"
Jul 21 18:05:17.587460 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.587439    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6b0cd43-b479-49e0-b918-f9ad25c44cd5" containerName="kube-rbac-proxy"
Jul 21 18:05:17.589601 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.589584    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:17.592228 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.592200    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:05:17.592534 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.592515    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:05:17.592793 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.592777    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:05:17.593620 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.593589    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:05:17.593721 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.593598    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 21 18:05:17.593721 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.593628    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 21 18:05:17.593975 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.593956    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:05:17.601488 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.601464    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"]
Jul 21 18:05:17.714456 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.714408    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvgmd\" (UniqueName: \"kubernetes.io/projected/3b2868fb-d23b-4ebc-a653-03df9fd62037-kube-api-access-fvgmd\") pod \"isvc-paddle-runtime-predictor-58b99fff76-6gl22\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:17.714670 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.714471    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b2868fb-d23b-4ebc-a653-03df9fd62037-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-6gl22\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:17.714670 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.714523    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b2868fb-d23b-4ebc-a653-03df9fd62037-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-6gl22\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:17.714670 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.714602    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b2868fb-d23b-4ebc-a653-03df9fd62037-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-6gl22\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:17.714670 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.714641    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b2868fb-d23b-4ebc-a653-03df9fd62037-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-6gl22\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:17.815875 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.815828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b2868fb-d23b-4ebc-a653-03df9fd62037-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-6gl22\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:17.816114 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.815892    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvgmd\" (UniqueName: \"kubernetes.io/projected/3b2868fb-d23b-4ebc-a653-03df9fd62037-kube-api-access-fvgmd\") pod \"isvc-paddle-runtime-predictor-58b99fff76-6gl22\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:17.816114 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.815921    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b2868fb-d23b-4ebc-a653-03df9fd62037-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-6gl22\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:17.816114 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.815940    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b2868fb-d23b-4ebc-a653-03df9fd62037-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-6gl22\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:17.816114 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.815967    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b2868fb-d23b-4ebc-a653-03df9fd62037-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-6gl22\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:17.816114 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:05:17.816112    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 21 18:05:17.816417 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:05:17.816183    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b2868fb-d23b-4ebc-a653-03df9fd62037-proxy-tls podName:3b2868fb-d23b-4ebc-a653-03df9fd62037 nodeName:}" failed. No retries permitted until 2026-07-21 18:05:18.316162095 +0000 UTC m=+2369.887783828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3b2868fb-d23b-4ebc-a653-03df9fd62037-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-6gl22" (UID: "3b2868fb-d23b-4ebc-a653-03df9fd62037") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 21 18:05:17.816496 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.816419    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b2868fb-d23b-4ebc-a653-03df9fd62037-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-6gl22\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:17.816698 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.816678    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b2868fb-d23b-4ebc-a653-03df9fd62037-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-6gl22\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:17.816698 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.816687    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b2868fb-d23b-4ebc-a653-03df9fd62037-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-6gl22\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:17.827692 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:17.827661    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvgmd\" (UniqueName: \"kubernetes.io/projected/3b2868fb-d23b-4ebc-a653-03df9fd62037-kube-api-access-fvgmd\") pod \"isvc-paddle-runtime-predictor-58b99fff76-6gl22\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:18.319685 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:18.319640    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b2868fb-d23b-4ebc-a653-03df9fd62037-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-6gl22\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:18.322368 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:18.322342    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b2868fb-d23b-4ebc-a653-03df9fd62037-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-6gl22\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:18.501055 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:18.501010    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:18.642834 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:18.642632    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"]
Jul 21 18:05:18.645273 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:05:18.645248    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2868fb_d23b_4ebc_a653_03df9fd62037.slice/crio-7db1e984c715b99eb91818611e6dcc11047e8461588b693dcebb5d736a939b73 WatchSource:0}: Error finding container 7db1e984c715b99eb91818611e6dcc11047e8461588b693dcebb5d736a939b73: Status 404 returned error can't find the container with id 7db1e984c715b99eb91818611e6dcc11047e8461588b693dcebb5d736a939b73
Jul 21 18:05:18.962155 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:18.962114    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" event={"ID":"3b2868fb-d23b-4ebc-a653-03df9fd62037","Type":"ContainerStarted","Data":"9798af8eac5bab9c794556737bb50e908c24856cbdfc7c29295835ec5b8cc9e9"}
Jul 21 18:05:18.962326 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:18.962183    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" event={"ID":"3b2868fb-d23b-4ebc-a653-03df9fd62037","Type":"ContainerStarted","Data":"7db1e984c715b99eb91818611e6dcc11047e8461588b693dcebb5d736a939b73"}
Jul 21 18:05:24.987549 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:24.987511    2562 generic.go:356] "Generic (PLEG): container finished" podID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerID="9798af8eac5bab9c794556737bb50e908c24856cbdfc7c29295835ec5b8cc9e9" exitCode=0
Jul 21 18:05:24.988009 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:24.987592    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" event={"ID":"3b2868fb-d23b-4ebc-a653-03df9fd62037","Type":"ContainerDied","Data":"9798af8eac5bab9c794556737bb50e908c24856cbdfc7c29295835ec5b8cc9e9"}
Jul 21 18:05:37.049760 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:37.049722    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" event={"ID":"3b2868fb-d23b-4ebc-a653-03df9fd62037","Type":"ContainerStarted","Data":"2306985b85dbf55e57934ff043c654071d92f664a59d62d63923ce655b860ce1"}
Jul 21 18:05:37.050329 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:37.049794    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" event={"ID":"3b2868fb-d23b-4ebc-a653-03df9fd62037","Type":"ContainerStarted","Data":"744399f997b6e6efab17059842a762b7cfab4b6c841ce3989b1341b24b65b36b"}
Jul 21 18:05:37.050329 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:37.050021    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:37.071776 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:37.071724    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" podStartSLOduration=3.289284269 podStartE2EDuration="20.071710252s" totalImagesPullingTime="11.418029443s" totalInitContainerRuntime="5.36439654s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:05:37.071251423 +0000 UTC m=+2388.642873179" watchObservedRunningTime="2026-07-21 18:05:37.071710252 +0000 UTC m=+2388.643332006"
Jul 21 18:05:38.053996 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:38.053967    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:38.055452 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:38.055420    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 21 18:05:39.058382 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:39.058343    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 21 18:05:44.063343 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:44.063314    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:05:44.063790 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:44.063764    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 21 18:05:49.013181 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:49.013146    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:05:49.015378 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:49.015355    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:05:54.063958 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:05:54.063917    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 21 18:06:04.063969 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:04.063928    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 21 18:06:14.064824 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:14.064734    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 21 18:06:24.064247 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:24.064214    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:06:29.276627 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:29.276594    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"]
Jul 21 18:06:29.277057 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:29.276809    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerName="kserve-container" containerID="cri-o://744399f997b6e6efab17059842a762b7cfab4b6c841ce3989b1341b24b65b36b" gracePeriod=30
Jul 21 18:06:29.277057 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:29.276846    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerName="kube-rbac-proxy" containerID="cri-o://2306985b85dbf55e57934ff043c654071d92f664a59d62d63923ce655b860ce1" gracePeriod=30
Jul 21 18:06:30.265703 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:30.265668    2562 generic.go:356] "Generic (PLEG): container finished" podID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerID="2306985b85dbf55e57934ff043c654071d92f664a59d62d63923ce655b860ce1" exitCode=2
Jul 21 18:06:30.265876 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:30.265712    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" event={"ID":"3b2868fb-d23b-4ebc-a653-03df9fd62037","Type":"ContainerDied","Data":"2306985b85dbf55e57934ff043c654071d92f664a59d62d63923ce655b860ce1"}
Jul 21 18:06:31.822247 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:31.822222    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:06:31.926705 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:31.926682    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b2868fb-d23b-4ebc-a653-03df9fd62037-cabundle-cert\") pod \"3b2868fb-d23b-4ebc-a653-03df9fd62037\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") "
Jul 21 18:06:31.926859 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:31.926755    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b2868fb-d23b-4ebc-a653-03df9fd62037-kserve-provision-location\") pod \"3b2868fb-d23b-4ebc-a653-03df9fd62037\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") "
Jul 21 18:06:31.926859 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:31.926781    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b2868fb-d23b-4ebc-a653-03df9fd62037-kube-rbac-proxy-sar-config\") pod \"3b2868fb-d23b-4ebc-a653-03df9fd62037\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") "
Jul 21 18:06:31.926859 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:31.926820    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b2868fb-d23b-4ebc-a653-03df9fd62037-proxy-tls\") pod \"3b2868fb-d23b-4ebc-a653-03df9fd62037\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") "
Jul 21 18:06:31.926859 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:31.926852    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvgmd\" (UniqueName: \"kubernetes.io/projected/3b2868fb-d23b-4ebc-a653-03df9fd62037-kube-api-access-fvgmd\") pod \"3b2868fb-d23b-4ebc-a653-03df9fd62037\" (UID: \"3b2868fb-d23b-4ebc-a653-03df9fd62037\") "
Jul 21 18:06:31.927135 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:31.927112    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2868fb-d23b-4ebc-a653-03df9fd62037-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3b2868fb-d23b-4ebc-a653-03df9fd62037" (UID: "3b2868fb-d23b-4ebc-a653-03df9fd62037"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:06:31.927201 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:31.927185    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2868fb-d23b-4ebc-a653-03df9fd62037-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3b2868fb-d23b-4ebc-a653-03df9fd62037" (UID: "3b2868fb-d23b-4ebc-a653-03df9fd62037"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:06:31.929142 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:31.929116    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2868fb-d23b-4ebc-a653-03df9fd62037-kube-api-access-fvgmd" (OuterVolumeSpecName: "kube-api-access-fvgmd") pod "3b2868fb-d23b-4ebc-a653-03df9fd62037" (UID: "3b2868fb-d23b-4ebc-a653-03df9fd62037"). InnerVolumeSpecName "kube-api-access-fvgmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:06:31.929142 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:31.929132    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2868fb-d23b-4ebc-a653-03df9fd62037-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3b2868fb-d23b-4ebc-a653-03df9fd62037" (UID: "3b2868fb-d23b-4ebc-a653-03df9fd62037"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:06:31.936667 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:31.936641    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2868fb-d23b-4ebc-a653-03df9fd62037-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b2868fb-d23b-4ebc-a653-03df9fd62037" (UID: "3b2868fb-d23b-4ebc-a653-03df9fd62037"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:06:32.027700 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.027672    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b2868fb-d23b-4ebc-a653-03df9fd62037-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:06:32.027816 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.027702    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvgmd\" (UniqueName: \"kubernetes.io/projected/3b2868fb-d23b-4ebc-a653-03df9fd62037-kube-api-access-fvgmd\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:06:32.027816 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.027716    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b2868fb-d23b-4ebc-a653-03df9fd62037-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:06:32.027816 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.027730    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b2868fb-d23b-4ebc-a653-03df9fd62037-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:06:32.027816 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.027743    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b2868fb-d23b-4ebc-a653-03df9fd62037-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:06:32.275016 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.274936    2562 generic.go:356] "Generic (PLEG): container finished" podID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerID="744399f997b6e6efab17059842a762b7cfab4b6c841ce3989b1341b24b65b36b" exitCode=0
Jul 21 18:06:32.275171 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.275010    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"
Jul 21 18:06:32.275171 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.275023    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" event={"ID":"3b2868fb-d23b-4ebc-a653-03df9fd62037","Type":"ContainerDied","Data":"744399f997b6e6efab17059842a762b7cfab4b6c841ce3989b1341b24b65b36b"}
Jul 21 18:06:32.275171 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.275100    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22" event={"ID":"3b2868fb-d23b-4ebc-a653-03df9fd62037","Type":"ContainerDied","Data":"7db1e984c715b99eb91818611e6dcc11047e8461588b693dcebb5d736a939b73"}
Jul 21 18:06:32.275171 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.275133    2562 scope.go:117] "RemoveContainer" containerID="2306985b85dbf55e57934ff043c654071d92f664a59d62d63923ce655b860ce1"
Jul 21 18:06:32.284616 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.284600    2562 scope.go:117] "RemoveContainer" containerID="744399f997b6e6efab17059842a762b7cfab4b6c841ce3989b1341b24b65b36b"
Jul 21 18:06:32.292322 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.292303    2562 scope.go:117] "RemoveContainer" containerID="9798af8eac5bab9c794556737bb50e908c24856cbdfc7c29295835ec5b8cc9e9"
Jul 21 18:06:32.300278 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.300170    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"]
Jul 21 18:06:32.300332 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.300311    2562 scope.go:117] "RemoveContainer" containerID="2306985b85dbf55e57934ff043c654071d92f664a59d62d63923ce655b860ce1"
Jul 21 18:06:32.300578 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:06:32.300559    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2306985b85dbf55e57934ff043c654071d92f664a59d62d63923ce655b860ce1\": container with ID starting with 2306985b85dbf55e57934ff043c654071d92f664a59d62d63923ce655b860ce1 not found: ID does not exist" containerID="2306985b85dbf55e57934ff043c654071d92f664a59d62d63923ce655b860ce1"
Jul 21 18:06:32.300626 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.300587    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2306985b85dbf55e57934ff043c654071d92f664a59d62d63923ce655b860ce1"} err="failed to get container status \"2306985b85dbf55e57934ff043c654071d92f664a59d62d63923ce655b860ce1\": rpc error: code = NotFound desc = could not find container \"2306985b85dbf55e57934ff043c654071d92f664a59d62d63923ce655b860ce1\": container with ID starting with 2306985b85dbf55e57934ff043c654071d92f664a59d62d63923ce655b860ce1 not found: ID does not exist"
Jul 21 18:06:32.300626 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.300607    2562 scope.go:117] "RemoveContainer" containerID="744399f997b6e6efab17059842a762b7cfab4b6c841ce3989b1341b24b65b36b"
Jul 21 18:06:32.300813 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:06:32.300791    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744399f997b6e6efab17059842a762b7cfab4b6c841ce3989b1341b24b65b36b\": container with ID starting with 744399f997b6e6efab17059842a762b7cfab4b6c841ce3989b1341b24b65b36b not found: ID does not exist" containerID="744399f997b6e6efab17059842a762b7cfab4b6c841ce3989b1341b24b65b36b"
Jul 21 18:06:32.300854 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.300818    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744399f997b6e6efab17059842a762b7cfab4b6c841ce3989b1341b24b65b36b"} err="failed to get container status \"744399f997b6e6efab17059842a762b7cfab4b6c841ce3989b1341b24b65b36b\": rpc error: code = NotFound desc = could not find container \"744399f997b6e6efab17059842a762b7cfab4b6c841ce3989b1341b24b65b36b\": container with ID starting with 744399f997b6e6efab17059842a762b7cfab4b6c841ce3989b1341b24b65b36b not found: ID does not exist"
Jul 21 18:06:32.300854 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.300847    2562 scope.go:117] "RemoveContainer" containerID="9798af8eac5bab9c794556737bb50e908c24856cbdfc7c29295835ec5b8cc9e9"
Jul 21 18:06:32.301076 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:06:32.301041    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9798af8eac5bab9c794556737bb50e908c24856cbdfc7c29295835ec5b8cc9e9\": container with ID starting with 9798af8eac5bab9c794556737bb50e908c24856cbdfc7c29295835ec5b8cc9e9 not found: ID does not exist" containerID="9798af8eac5bab9c794556737bb50e908c24856cbdfc7c29295835ec5b8cc9e9"
Jul 21 18:06:32.301134 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.301086    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9798af8eac5bab9c794556737bb50e908c24856cbdfc7c29295835ec5b8cc9e9"} err="failed to get container status \"9798af8eac5bab9c794556737bb50e908c24856cbdfc7c29295835ec5b8cc9e9\": rpc error: code = NotFound desc = could not find container \"9798af8eac5bab9c794556737bb50e908c24856cbdfc7c29295835ec5b8cc9e9\": container with ID starting with 9798af8eac5bab9c794556737bb50e908c24856cbdfc7c29295835ec5b8cc9e9 not found: ID does not exist"
Jul 21 18:06:32.303595 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.303576    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-6gl22"]
Jul 21 18:06:32.921370 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:06:32.921339    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" path="/var/lib/kubelet/pods/3b2868fb-d23b-4ebc-a653-03df9fd62037/volumes"
Jul 21 18:09:37.796035 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.795996    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"]
Jul 21 18:09:37.796548 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.796344    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerName="storage-initializer"
Jul 21 18:09:37.796548 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.796357    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerName="storage-initializer"
Jul 21 18:09:37.796548 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.796368    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerName="kube-rbac-proxy"
Jul 21 18:09:37.796548 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.796374    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerName="kube-rbac-proxy"
Jul 21 18:09:37.796548 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.796384    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerName="kserve-container"
Jul 21 18:09:37.796548 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.796390    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerName="kserve-container"
Jul 21 18:09:37.796548 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.796450    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerName="kserve-container"
Jul 21 18:09:37.796548 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.796470    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b2868fb-d23b-4ebc-a653-03df9fd62037" containerName="kube-rbac-proxy"
Jul 21 18:09:37.799523 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.799507    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:37.802166 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.802143    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 21 18:09:37.802309 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.802197    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:09:37.802309 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.802219    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:09:37.802500 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.802484    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:09:37.803354 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.803333    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 21 18:09:37.803456 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.803369    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:09:37.803456 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.803434    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:09:37.810991 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.810969    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"]
Jul 21 18:09:37.921957 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.921924    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bad146a-9928-4ec2-91b5-146ab68c9758-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-s42k4\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:37.921957 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.921958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bad146a-9928-4ec2-91b5-146ab68c9758-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-s42k4\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:37.922199 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.921980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bad146a-9928-4ec2-91b5-146ab68c9758-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-s42k4\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:37.922199 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.922048    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwcng\" (UniqueName: \"kubernetes.io/projected/8bad146a-9928-4ec2-91b5-146ab68c9758-kube-api-access-mwcng\") pod \"isvc-pmml-runtime-predictor-885fc8777-s42k4\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:37.922199 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:37.922088    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bad146a-9928-4ec2-91b5-146ab68c9758-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-s42k4\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:38.023350 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:38.023320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bad146a-9928-4ec2-91b5-146ab68c9758-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-s42k4\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:38.023350 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:38.023350    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bad146a-9928-4ec2-91b5-146ab68c9758-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-s42k4\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:38.023580 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:38.023372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bad146a-9928-4ec2-91b5-146ab68c9758-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-s42k4\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:38.023580 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:38.023417    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwcng\" (UniqueName: \"kubernetes.io/projected/8bad146a-9928-4ec2-91b5-146ab68c9758-kube-api-access-mwcng\") pod \"isvc-pmml-runtime-predictor-885fc8777-s42k4\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:38.023580 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:38.023439    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bad146a-9928-4ec2-91b5-146ab68c9758-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-s42k4\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:38.023580 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:09:38.023521    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
Jul 21 18:09:38.023743 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:09:38.023598    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bad146a-9928-4ec2-91b5-146ab68c9758-proxy-tls podName:8bad146a-9928-4ec2-91b5-146ab68c9758 nodeName:}" failed. No retries permitted until 2026-07-21 18:09:38.52357548 +0000 UTC m=+2630.095197213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8bad146a-9928-4ec2-91b5-146ab68c9758-proxy-tls") pod "isvc-pmml-runtime-predictor-885fc8777-s42k4" (UID: "8bad146a-9928-4ec2-91b5-146ab68c9758") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
Jul 21 18:09:38.023851 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:38.023835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bad146a-9928-4ec2-91b5-146ab68c9758-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-s42k4\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:38.023977 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:38.023960    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bad146a-9928-4ec2-91b5-146ab68c9758-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-s42k4\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:38.024096 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:38.024051    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bad146a-9928-4ec2-91b5-146ab68c9758-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-s42k4\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:38.032974 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:38.032946    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwcng\" (UniqueName: \"kubernetes.io/projected/8bad146a-9928-4ec2-91b5-146ab68c9758-kube-api-access-mwcng\") pod \"isvc-pmml-runtime-predictor-885fc8777-s42k4\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:38.527982 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:38.527948    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bad146a-9928-4ec2-91b5-146ab68c9758-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-s42k4\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:38.530514 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:38.530484    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bad146a-9928-4ec2-91b5-146ab68c9758-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-s42k4\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:38.710222 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:38.710179    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:38.849876 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:38.849855    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"]
Jul 21 18:09:38.851846 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:09:38.851816    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bad146a_9928_4ec2_91b5_146ab68c9758.slice/crio-ff5e0d6a59737f91ab795a043f63a1dfcec8de4e7e863517ea05b79e89a79f31 WatchSource:0}: Error finding container ff5e0d6a59737f91ab795a043f63a1dfcec8de4e7e863517ea05b79e89a79f31: Status 404 returned error can't find the container with id ff5e0d6a59737f91ab795a043f63a1dfcec8de4e7e863517ea05b79e89a79f31
Jul 21 18:09:39.016948 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:39.016912    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" event={"ID":"8bad146a-9928-4ec2-91b5-146ab68c9758","Type":"ContainerStarted","Data":"5310796b1369e6da36895e713e712249b45992bb9a01199bf7f5a90b61ce4029"}
Jul 21 18:09:39.017167 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:39.016972    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" event={"ID":"8bad146a-9928-4ec2-91b5-146ab68c9758","Type":"ContainerStarted","Data":"ff5e0d6a59737f91ab795a043f63a1dfcec8de4e7e863517ea05b79e89a79f31"}
Jul 21 18:09:43.033366 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:43.033332    2562 generic.go:356] "Generic (PLEG): container finished" podID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerID="5310796b1369e6da36895e713e712249b45992bb9a01199bf7f5a90b61ce4029" exitCode=0
Jul 21 18:09:43.033886 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:43.033410    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" event={"ID":"8bad146a-9928-4ec2-91b5-146ab68c9758","Type":"ContainerDied","Data":"5310796b1369e6da36895e713e712249b45992bb9a01199bf7f5a90b61ce4029"}
Jul 21 18:09:43.034734 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:43.034720    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 18:09:50.070244 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:50.070209    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" event={"ID":"8bad146a-9928-4ec2-91b5-146ab68c9758","Type":"ContainerStarted","Data":"205eda0889e4425b8130e3951e548ab83bc5e20fdcd0dd9cf7f93e9047dac910"}
Jul 21 18:09:50.070652 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:50.070282    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" event={"ID":"8bad146a-9928-4ec2-91b5-146ab68c9758","Type":"ContainerStarted","Data":"1587c6b77aad95b83a8f07d82859fcb29d427fa61013a1364e4a33224acdac24"}
Jul 21 18:09:50.070652 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:50.070581    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:50.070652 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:50.070625    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:50.071975 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:50.071949    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 21 18:09:51.074056 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:51.074015    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 21 18:09:56.079246 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:56.079216    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:09:56.079719 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:56.079695    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 21 18:09:56.102377 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:09:56.102330    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" podStartSLOduration=8.813869704 podStartE2EDuration="19.102318576s" totalImagesPullingTime="6.86452844s" totalInitContainerRuntime="3.423920432s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:09:50.092796232 +0000 UTC m=+2641.664418035" watchObservedRunningTime="2026-07-21 18:09:56.102318576 +0000 UTC m=+2647.673940331"
Jul 21 18:10:06.079864 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:10:06.079823    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 21 18:10:16.079759 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:10:16.079718    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 21 18:10:26.079798 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:10:26.079758    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 21 18:10:36.080421 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:10:36.080367    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 21 18:10:46.080448 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:10:46.080406    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 21 18:10:49.046096 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:10:49.046052    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:10:49.049386 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:10:49.049365    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:10:56.079988 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:10:56.079951    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 21 18:11:06.080179 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:06.080138    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 21 18:11:16.080793 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:16.080758    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:11:18.973614 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:18.973580    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"]
Jul 21 18:11:18.974117 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:18.973908    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kserve-container" containerID="cri-o://1587c6b77aad95b83a8f07d82859fcb29d427fa61013a1364e4a33224acdac24" gracePeriod=30
Jul 21 18:11:18.974117 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:18.973949    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kube-rbac-proxy" containerID="cri-o://205eda0889e4425b8130e3951e548ab83bc5e20fdcd0dd9cf7f93e9047dac910" gracePeriod=30
Jul 21 18:11:19.429637 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:19.429606    2562 generic.go:356] "Generic (PLEG): container finished" podID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerID="205eda0889e4425b8130e3951e548ab83bc5e20fdcd0dd9cf7f93e9047dac910" exitCode=2
Jul 21 18:11:19.429809 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:19.429680    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" event={"ID":"8bad146a-9928-4ec2-91b5-146ab68c9758","Type":"ContainerDied","Data":"205eda0889e4425b8130e3951e548ab83bc5e20fdcd0dd9cf7f93e9047dac910"}
Jul 21 18:11:21.075124 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:21.075086    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 21 18:11:22.421633 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.421610    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:11:22.444099 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.444049    2562 generic.go:356] "Generic (PLEG): container finished" podID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerID="1587c6b77aad95b83a8f07d82859fcb29d427fa61013a1364e4a33224acdac24" exitCode=0
Jul 21 18:11:22.444245 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.444103    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" event={"ID":"8bad146a-9928-4ec2-91b5-146ab68c9758","Type":"ContainerDied","Data":"1587c6b77aad95b83a8f07d82859fcb29d427fa61013a1364e4a33224acdac24"}
Jul 21 18:11:22.444245 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.444133    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"
Jul 21 18:11:22.444245 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.444162    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4" event={"ID":"8bad146a-9928-4ec2-91b5-146ab68c9758","Type":"ContainerDied","Data":"ff5e0d6a59737f91ab795a043f63a1dfcec8de4e7e863517ea05b79e89a79f31"}
Jul 21 18:11:22.444245 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.444191    2562 scope.go:117] "RemoveContainer" containerID="205eda0889e4425b8130e3951e548ab83bc5e20fdcd0dd9cf7f93e9047dac910"
Jul 21 18:11:22.454741 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.454723    2562 scope.go:117] "RemoveContainer" containerID="1587c6b77aad95b83a8f07d82859fcb29d427fa61013a1364e4a33224acdac24"
Jul 21 18:11:22.463993 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.463971    2562 scope.go:117] "RemoveContainer" containerID="5310796b1369e6da36895e713e712249b45992bb9a01199bf7f5a90b61ce4029"
Jul 21 18:11:22.471688 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.471671    2562 scope.go:117] "RemoveContainer" containerID="205eda0889e4425b8130e3951e548ab83bc5e20fdcd0dd9cf7f93e9047dac910"
Jul 21 18:11:22.471966 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:11:22.471949    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205eda0889e4425b8130e3951e548ab83bc5e20fdcd0dd9cf7f93e9047dac910\": container with ID starting with 205eda0889e4425b8130e3951e548ab83bc5e20fdcd0dd9cf7f93e9047dac910 not found: ID does not exist" containerID="205eda0889e4425b8130e3951e548ab83bc5e20fdcd0dd9cf7f93e9047dac910"
Jul 21 18:11:22.472014 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.471978    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205eda0889e4425b8130e3951e548ab83bc5e20fdcd0dd9cf7f93e9047dac910"} err="failed to get container status \"205eda0889e4425b8130e3951e548ab83bc5e20fdcd0dd9cf7f93e9047dac910\": rpc error: code = NotFound desc = could not find container \"205eda0889e4425b8130e3951e548ab83bc5e20fdcd0dd9cf7f93e9047dac910\": container with ID starting with 205eda0889e4425b8130e3951e548ab83bc5e20fdcd0dd9cf7f93e9047dac910 not found: ID does not exist"
Jul 21 18:11:22.472014 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.472001    2562 scope.go:117] "RemoveContainer" containerID="1587c6b77aad95b83a8f07d82859fcb29d427fa61013a1364e4a33224acdac24"
Jul 21 18:11:22.472270 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:11:22.472253    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1587c6b77aad95b83a8f07d82859fcb29d427fa61013a1364e4a33224acdac24\": container with ID starting with 1587c6b77aad95b83a8f07d82859fcb29d427fa61013a1364e4a33224acdac24 not found: ID does not exist" containerID="1587c6b77aad95b83a8f07d82859fcb29d427fa61013a1364e4a33224acdac24"
Jul 21 18:11:22.472329 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.472274    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1587c6b77aad95b83a8f07d82859fcb29d427fa61013a1364e4a33224acdac24"} err="failed to get container status \"1587c6b77aad95b83a8f07d82859fcb29d427fa61013a1364e4a33224acdac24\": rpc error: code = NotFound desc = could not find container \"1587c6b77aad95b83a8f07d82859fcb29d427fa61013a1364e4a33224acdac24\": container with ID starting with 1587c6b77aad95b83a8f07d82859fcb29d427fa61013a1364e4a33224acdac24 not found: ID does not exist"
Jul 21 18:11:22.472329 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.472290    2562 scope.go:117] "RemoveContainer" containerID="5310796b1369e6da36895e713e712249b45992bb9a01199bf7f5a90b61ce4029"
Jul 21 18:11:22.472544 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:11:22.472524    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5310796b1369e6da36895e713e712249b45992bb9a01199bf7f5a90b61ce4029\": container with ID starting with 5310796b1369e6da36895e713e712249b45992bb9a01199bf7f5a90b61ce4029 not found: ID does not exist" containerID="5310796b1369e6da36895e713e712249b45992bb9a01199bf7f5a90b61ce4029"
Jul 21 18:11:22.472582 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.472552    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5310796b1369e6da36895e713e712249b45992bb9a01199bf7f5a90b61ce4029"} err="failed to get container status \"5310796b1369e6da36895e713e712249b45992bb9a01199bf7f5a90b61ce4029\": rpc error: code = NotFound desc = could not find container \"5310796b1369e6da36895e713e712249b45992bb9a01199bf7f5a90b61ce4029\": container with ID starting with 5310796b1369e6da36895e713e712249b45992bb9a01199bf7f5a90b61ce4029 not found: ID does not exist"
Jul 21 18:11:22.475848 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.475825    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bad146a-9928-4ec2-91b5-146ab68c9758-cabundle-cert\") pod \"8bad146a-9928-4ec2-91b5-146ab68c9758\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") "
Jul 21 18:11:22.475912 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.475880    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bad146a-9928-4ec2-91b5-146ab68c9758-kube-rbac-proxy-sar-config\") pod \"8bad146a-9928-4ec2-91b5-146ab68c9758\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") "
Jul 21 18:11:22.475945 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.475914    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwcng\" (UniqueName: \"kubernetes.io/projected/8bad146a-9928-4ec2-91b5-146ab68c9758-kube-api-access-mwcng\") pod \"8bad146a-9928-4ec2-91b5-146ab68c9758\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") "
Jul 21 18:11:22.475978 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.475952    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bad146a-9928-4ec2-91b5-146ab68c9758-proxy-tls\") pod \"8bad146a-9928-4ec2-91b5-146ab68c9758\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") "
Jul 21 18:11:22.476084 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.476045    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bad146a-9928-4ec2-91b5-146ab68c9758-kserve-provision-location\") pod \"8bad146a-9928-4ec2-91b5-146ab68c9758\" (UID: \"8bad146a-9928-4ec2-91b5-146ab68c9758\") "
Jul 21 18:11:22.476271 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.476238    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bad146a-9928-4ec2-91b5-146ab68c9758-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8bad146a-9928-4ec2-91b5-146ab68c9758" (UID: "8bad146a-9928-4ec2-91b5-146ab68c9758"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:11:22.476400 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.476296    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bad146a-9928-4ec2-91b5-146ab68c9758-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8bad146a-9928-4ec2-91b5-146ab68c9758" (UID: "8bad146a-9928-4ec2-91b5-146ab68c9758"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:11:22.476536 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.476406    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bad146a-9928-4ec2-91b5-146ab68c9758-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:11:22.476536 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.476424    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bad146a-9928-4ec2-91b5-146ab68c9758-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:11:22.476646 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.476581    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bad146a-9928-4ec2-91b5-146ab68c9758-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8bad146a-9928-4ec2-91b5-146ab68c9758" (UID: "8bad146a-9928-4ec2-91b5-146ab68c9758"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:11:22.478226 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.478203    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bad146a-9928-4ec2-91b5-146ab68c9758-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8bad146a-9928-4ec2-91b5-146ab68c9758" (UID: "8bad146a-9928-4ec2-91b5-146ab68c9758"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:11:22.478302 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.478279    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bad146a-9928-4ec2-91b5-146ab68c9758-kube-api-access-mwcng" (OuterVolumeSpecName: "kube-api-access-mwcng") pod "8bad146a-9928-4ec2-91b5-146ab68c9758" (UID: "8bad146a-9928-4ec2-91b5-146ab68c9758"). InnerVolumeSpecName "kube-api-access-mwcng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:11:22.577408 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.577373    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwcng\" (UniqueName: \"kubernetes.io/projected/8bad146a-9928-4ec2-91b5-146ab68c9758-kube-api-access-mwcng\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:11:22.577408 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.577401    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bad146a-9928-4ec2-91b5-146ab68c9758-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:11:22.577408 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.577412    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bad146a-9928-4ec2-91b5-146ab68c9758-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:11:22.766271 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.766245    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"]
Jul 21 18:11:22.769907 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.769882    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-s42k4"]
Jul 21 18:11:22.921943 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:11:22.921913    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" path="/var/lib/kubelet/pods/8bad146a-9928-4ec2-91b5-146ab68c9758/volumes"
Jul 21 18:13:00.592739 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.592702    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"]
Jul 21 18:13:00.593210 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.593004    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kserve-container"
Jul 21 18:13:00.593210 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.593014    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kserve-container"
Jul 21 18:13:00.593210 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.593033    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="storage-initializer"
Jul 21 18:13:00.593210 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.593040    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="storage-initializer"
Jul 21 18:13:00.593210 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.593049    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kube-rbac-proxy"
Jul 21 18:13:00.593210 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.593055    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kube-rbac-proxy"
Jul 21 18:13:00.593210 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.593134    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kserve-container"
Jul 21 18:13:00.593210 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.593142    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bad146a-9928-4ec2-91b5-146ab68c9758" containerName="kube-rbac-proxy"
Jul 21 18:13:00.594972 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.594958    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:00.597760 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.597735    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:13:00.597912 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.597785    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-6e1ef9-kube-rbac-proxy-sar-config\""
Jul 21 18:13:00.597912 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.597735    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:13:00.597912 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.597879    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:13:00.598138 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.598018    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:13:00.598932 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.598915    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:13:00.599027 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.599001    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-6e1ef9-predictor-serving-cert\""
Jul 21 18:13:00.606965 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.606929    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"]
Jul 21 18:13:00.778470 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.778430    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b60acdd-163f-48a3-a24e-c33cb88fe093-kserve-provision-location\") pod \"isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") " pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:00.778470 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.778470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b60acdd-163f-48a3-a24e-c33cb88fe093-cabundle-cert\") pod \"isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") " pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:00.778703 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.778545    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b60acdd-163f-48a3-a24e-c33cb88fe093-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") " pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:00.778703 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.778583    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b60acdd-163f-48a3-a24e-c33cb88fe093-proxy-tls\") pod \"isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") " pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:00.778703 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.778623    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkzfs\" (UniqueName: \"kubernetes.io/projected/7b60acdd-163f-48a3-a24e-c33cb88fe093-kube-api-access-bkzfs\") pod \"isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") " pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:00.879139 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.879024    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b60acdd-163f-48a3-a24e-c33cb88fe093-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") " pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:00.879139 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.879097    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b60acdd-163f-48a3-a24e-c33cb88fe093-proxy-tls\") pod \"isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") " pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:00.879380 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.879145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkzfs\" (UniqueName: \"kubernetes.io/projected/7b60acdd-163f-48a3-a24e-c33cb88fe093-kube-api-access-bkzfs\") pod \"isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") " pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:00.879380 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.879200    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b60acdd-163f-48a3-a24e-c33cb88fe093-kserve-provision-location\") pod \"isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") " pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:00.879380 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.879225    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b60acdd-163f-48a3-a24e-c33cb88fe093-cabundle-cert\") pod \"isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") " pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:00.879545 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.879507    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b60acdd-163f-48a3-a24e-c33cb88fe093-kserve-provision-location\") pod \"isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") " pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:00.879724 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.879701    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b60acdd-163f-48a3-a24e-c33cb88fe093-kube-rbac-proxy-sar-config\") pod \"isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") " pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:00.879849 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.879830    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b60acdd-163f-48a3-a24e-c33cb88fe093-cabundle-cert\") pod \"isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") " pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:00.881688 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.881658    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b60acdd-163f-48a3-a24e-c33cb88fe093-proxy-tls\") pod \"isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") " pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:00.887351 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.887329    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkzfs\" (UniqueName: \"kubernetes.io/projected/7b60acdd-163f-48a3-a24e-c33cb88fe093-kube-api-access-bkzfs\") pod \"isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") " pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:00.905182 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:00.905157    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:01.037239 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:01.037178    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"]
Jul 21 18:13:01.039654 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:13:01.039626    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b60acdd_163f_48a3_a24e_c33cb88fe093.slice/crio-b2b87498fb64ee05b925e63c6a64a6df2182488416d3a20b670839fad4da9496 WatchSource:0}: Error finding container b2b87498fb64ee05b925e63c6a64a6df2182488416d3a20b670839fad4da9496: Status 404 returned error can't find the container with id b2b87498fb64ee05b925e63c6a64a6df2182488416d3a20b670839fad4da9496
Jul 21 18:13:01.840937 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:01.840900    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" event={"ID":"7b60acdd-163f-48a3-a24e-c33cb88fe093","Type":"ContainerStarted","Data":"aeff6080822faa6e08bcdee19c34f7423a345c0065c6484924a7839f395579a0"}
Jul 21 18:13:01.841409 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:01.840956    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" event={"ID":"7b60acdd-163f-48a3-a24e-c33cb88fe093","Type":"ContainerStarted","Data":"b2b87498fb64ee05b925e63c6a64a6df2182488416d3a20b670839fad4da9496"}
Jul 21 18:13:04.853699 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:04.853664    2562 generic.go:356] "Generic (PLEG): container finished" podID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerID="aeff6080822faa6e08bcdee19c34f7423a345c0065c6484924a7839f395579a0" exitCode=0
Jul 21 18:13:04.854168 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:04.853739    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" event={"ID":"7b60acdd-163f-48a3-a24e-c33cb88fe093","Type":"ContainerDied","Data":"aeff6080822faa6e08bcdee19c34f7423a345c0065c6484924a7839f395579a0"}
Jul 21 18:13:05.860511 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:05.860471    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" event={"ID":"7b60acdd-163f-48a3-a24e-c33cb88fe093","Type":"ContainerStarted","Data":"ed43d2a2e996cdf4bb2aa7c504973385054fa3b2a0409a4b4ad740aa19303b35"}
Jul 21 18:13:05.860996 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:05.860554    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" event={"ID":"7b60acdd-163f-48a3-a24e-c33cb88fe093","Type":"ContainerStarted","Data":"f9b2b4dbe36053e68c4b98b3e3a7a88f44f0b196a1ed5ae776c525d7f30e4bf4"}
Jul 21 18:13:05.860996 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:05.860829    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:05.860996 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:05.860959    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:05.862216 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:05.862189    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 21 18:13:05.882489 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:05.882425    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" podStartSLOduration=2.279985891 podStartE2EDuration="5.88240227s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.602416379s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:13:05.881472663 +0000 UTC m=+2837.453094441" watchObservedRunningTime="2026-07-21 18:13:05.88240227 +0000 UTC m=+2837.454024024"
Jul 21 18:13:06.864693 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:06.864655    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 21 18:13:11.869142 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:11.869106    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:13:11.869780 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:11.869753    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 21 18:13:21.870276 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:21.870226    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 21 18:13:31.870546 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:31.870504    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 21 18:13:41.870137 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:41.870102    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 21 18:13:51.871240 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:13:51.871192    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 21 18:14:01.869931 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:01.869885    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 21 18:14:11.870144 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:11.870102    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 21 18:14:21.870658 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:21.870627    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:14:45.806526 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:45.806484    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"]
Jul 21 18:14:45.809275 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:45.809253    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:45.812011 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:45.811988    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-6e1ef9-predictor-serving-cert\""
Jul 21 18:14:45.812150 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:45.812007    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-6e1ef9-dockercfg-xf59t\""
Jul 21 18:14:45.812150 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:45.811995    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-6e1ef9\""
Jul 21 18:14:45.812150 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:45.812058    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-6e1ef9-kube-rbac-proxy-sar-config\""
Jul 21 18:14:45.819791 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:45.819772    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"]
Jul 21 18:14:45.934751 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:45.934723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knvfp\" (UniqueName: \"kubernetes.io/projected/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kube-api-access-knvfp\") pod \"isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") " pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:45.934879 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:45.934760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b3a0d6-6486-4bff-a78d-c6ac5b265946-proxy-tls\") pod \"isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") " pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:45.934879 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:45.934798    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80b3a0d6-6486-4bff-a78d-c6ac5b265946-cabundle-cert\") pod \"isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") " pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:45.934879 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:45.934833    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") " pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:45.934879 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:45.934859    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kserve-provision-location\") pod \"isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") " pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:46.035998 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:46.035967    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") " pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:46.036160 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:46.036038    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kserve-provision-location\") pod \"isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") " pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:46.036160 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:46.036105    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knvfp\" (UniqueName: \"kubernetes.io/projected/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kube-api-access-knvfp\") pod \"isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") " pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:46.036160 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:46.036143    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b3a0d6-6486-4bff-a78d-c6ac5b265946-proxy-tls\") pod \"isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") " pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:46.036324 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:46.036187    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80b3a0d6-6486-4bff-a78d-c6ac5b265946-cabundle-cert\") pod \"isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") " pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:46.036390 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:14:46.036375    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-serving-cert: secret "isvc-secondary-6e1ef9-predictor-serving-cert" not found
Jul 21 18:14:46.036462 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:14:46.036445    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80b3a0d6-6486-4bff-a78d-c6ac5b265946-proxy-tls podName:80b3a0d6-6486-4bff-a78d-c6ac5b265946 nodeName:}" failed. No retries permitted until 2026-07-21 18:14:46.536423599 +0000 UTC m=+2938.108045338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/80b3a0d6-6486-4bff-a78d-c6ac5b265946-proxy-tls") pod "isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8" (UID: "80b3a0d6-6486-4bff-a78d-c6ac5b265946") : secret "isvc-secondary-6e1ef9-predictor-serving-cert" not found
Jul 21 18:14:46.036744 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:46.036724    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kserve-provision-location\") pod \"isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") " pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:46.036904 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:46.036884    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") " pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:46.036976 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:46.036962    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80b3a0d6-6486-4bff-a78d-c6ac5b265946-cabundle-cert\") pod \"isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") " pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:46.044845 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:46.044826    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knvfp\" (UniqueName: \"kubernetes.io/projected/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kube-api-access-knvfp\") pod \"isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") " pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:46.539751 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:46.539721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b3a0d6-6486-4bff-a78d-c6ac5b265946-proxy-tls\") pod \"isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") " pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:46.542220 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:46.542201    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b3a0d6-6486-4bff-a78d-c6ac5b265946-proxy-tls\") pod \"isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") " pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:46.719746 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:46.719713    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:14:46.853951 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:46.853921    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"]
Jul 21 18:14:46.857432 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:14:46.857405    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b3a0d6_6486_4bff_a78d_c6ac5b265946.slice/crio-bc399a2adb1c626bb004b146440fa481ec74571e7d0d60ac82d75c6b4e09d2b4 WatchSource:0}: Error finding container bc399a2adb1c626bb004b146440fa481ec74571e7d0d60ac82d75c6b4e09d2b4: Status 404 returned error can't find the container with id bc399a2adb1c626bb004b146440fa481ec74571e7d0d60ac82d75c6b4e09d2b4
Jul 21 18:14:47.273373 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:47.273337    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8" event={"ID":"80b3a0d6-6486-4bff-a78d-c6ac5b265946","Type":"ContainerStarted","Data":"ceb2d35e6e27a4e63e4596b894e690192da1b568b9f9d288947f59c09f487db3"}
Jul 21 18:14:47.273552 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:47.273420    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8" event={"ID":"80b3a0d6-6486-4bff-a78d-c6ac5b265946","Type":"ContainerStarted","Data":"bc399a2adb1c626bb004b146440fa481ec74571e7d0d60ac82d75c6b4e09d2b4"}
Jul 21 18:14:50.291059 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:50.291031    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8_80b3a0d6-6486-4bff-a78d-c6ac5b265946/storage-initializer/0.log"
Jul 21 18:14:50.291457 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:50.291090    2562 generic.go:356] "Generic (PLEG): container finished" podID="80b3a0d6-6486-4bff-a78d-c6ac5b265946" containerID="ceb2d35e6e27a4e63e4596b894e690192da1b568b9f9d288947f59c09f487db3" exitCode=1
Jul 21 18:14:50.291457 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:50.291157    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8" event={"ID":"80b3a0d6-6486-4bff-a78d-c6ac5b265946","Type":"ContainerDied","Data":"ceb2d35e6e27a4e63e4596b894e690192da1b568b9f9d288947f59c09f487db3"}
Jul 21 18:14:51.297384 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:51.297357    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8_80b3a0d6-6486-4bff-a78d-c6ac5b265946/storage-initializer/0.log"
Jul 21 18:14:51.297851 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:51.297422    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8" event={"ID":"80b3a0d6-6486-4bff-a78d-c6ac5b265946","Type":"ContainerStarted","Data":"5e8bd71f7a4efae5091d2a8999dc95b23e77be9120f1a7d947a2e0c75012bbf7"}
Jul 21 18:14:56.320533 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:56.320492    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8_80b3a0d6-6486-4bff-a78d-c6ac5b265946/storage-initializer/1.log"
Jul 21 18:14:56.320926 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:56.320888    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8_80b3a0d6-6486-4bff-a78d-c6ac5b265946/storage-initializer/0.log"
Jul 21 18:14:56.320972 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:56.320920    2562 generic.go:356] "Generic (PLEG): container finished" podID="80b3a0d6-6486-4bff-a78d-c6ac5b265946" containerID="5e8bd71f7a4efae5091d2a8999dc95b23e77be9120f1a7d947a2e0c75012bbf7" exitCode=1
Jul 21 18:14:56.321021 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:56.320977    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8" event={"ID":"80b3a0d6-6486-4bff-a78d-c6ac5b265946","Type":"ContainerDied","Data":"5e8bd71f7a4efae5091d2a8999dc95b23e77be9120f1a7d947a2e0c75012bbf7"}
Jul 21 18:14:56.321092 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:56.321055    2562 scope.go:117] "RemoveContainer" containerID="ceb2d35e6e27a4e63e4596b894e690192da1b568b9f9d288947f59c09f487db3"
Jul 21 18:14:56.321440 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:56.321419    2562 scope.go:117] "RemoveContainer" containerID="ceb2d35e6e27a4e63e4596b894e690192da1b568b9f9d288947f59c09f487db3"
Jul 21 18:14:56.331396 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:14:56.331373    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8_kserve-ci-e2e-test(80b3a0d6-6486-4bff-a78d-c6ac5b265946)\"" pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8" podUID="80b3a0d6-6486-4bff-a78d-c6ac5b265946"
Jul 21 18:14:56.331693 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:14:56.331670    2562 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8_kserve-ci-e2e-test_80b3a0d6-6486-4bff-a78d-c6ac5b265946_0 in pod sandbox bc399a2adb1c626bb004b146440fa481ec74571e7d0d60ac82d75c6b4e09d2b4 from index: no such id: 'ceb2d35e6e27a4e63e4596b894e690192da1b568b9f9d288947f59c09f487db3'" containerID="ceb2d35e6e27a4e63e4596b894e690192da1b568b9f9d288947f59c09f487db3"
Jul 21 18:14:56.331743 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:56.331701    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb2d35e6e27a4e63e4596b894e690192da1b568b9f9d288947f59c09f487db3"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8_kserve-ci-e2e-test_80b3a0d6-6486-4bff-a78d-c6ac5b265946_0 in pod sandbox bc399a2adb1c626bb004b146440fa481ec74571e7d0d60ac82d75c6b4e09d2b4 from index: no such id: 'ceb2d35e6e27a4e63e4596b894e690192da1b568b9f9d288947f59c09f487db3'"
Jul 21 18:14:57.327681 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:14:57.327656    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8_80b3a0d6-6486-4bff-a78d-c6ac5b265946/storage-initializer/1.log"
Jul 21 18:15:01.850021 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:01.849983    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"]
Jul 21 18:15:01.910394 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:01.910363    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"]
Jul 21 18:15:01.910723 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:01.910671    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kserve-container" containerID="cri-o://f9b2b4dbe36053e68c4b98b3e3a7a88f44f0b196a1ed5ae776c525d7f30e4bf4" gracePeriod=30
Jul 21 18:15:01.910848 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:01.910739    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kube-rbac-proxy" containerID="cri-o://ed43d2a2e996cdf4bb2aa7c504973385054fa3b2a0409a4b4ad740aa19303b35" gracePeriod=30
Jul 21 18:15:02.031433 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.031413    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8_80b3a0d6-6486-4bff-a78d-c6ac5b265946/storage-initializer/1.log"
Jul 21 18:15:02.033490 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.033474    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:15:02.167943 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.167918    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knvfp\" (UniqueName: \"kubernetes.io/projected/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kube-api-access-knvfp\") pod \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") "
Jul 21 18:15:02.168137 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.167969    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b3a0d6-6486-4bff-a78d-c6ac5b265946-proxy-tls\") pod \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") "
Jul 21 18:15:02.168137 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.167997    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80b3a0d6-6486-4bff-a78d-c6ac5b265946-cabundle-cert\") pod \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") "
Jul 21 18:15:02.168137 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.168032    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kserve-provision-location\") pod \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") "
Jul 21 18:15:02.168137 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.168103    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kube-rbac-proxy-sar-config\") pod \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\" (UID: \"80b3a0d6-6486-4bff-a78d-c6ac5b265946\") "
Jul 21 18:15:02.168406 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.168355    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "80b3a0d6-6486-4bff-a78d-c6ac5b265946" (UID: "80b3a0d6-6486-4bff-a78d-c6ac5b265946"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:15:02.168472 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.168446    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b3a0d6-6486-4bff-a78d-c6ac5b265946-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "80b3a0d6-6486-4bff-a78d-c6ac5b265946" (UID: "80b3a0d6-6486-4bff-a78d-c6ac5b265946"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:15:02.168520 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.168481    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "80b3a0d6-6486-4bff-a78d-c6ac5b265946" (UID: "80b3a0d6-6486-4bff-a78d-c6ac5b265946"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:15:02.170179 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.170157    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kube-api-access-knvfp" (OuterVolumeSpecName: "kube-api-access-knvfp") pod "80b3a0d6-6486-4bff-a78d-c6ac5b265946" (UID: "80b3a0d6-6486-4bff-a78d-c6ac5b265946"). InnerVolumeSpecName "kube-api-access-knvfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:15:02.170305 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.170281    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b3a0d6-6486-4bff-a78d-c6ac5b265946-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "80b3a0d6-6486-4bff-a78d-c6ac5b265946" (UID: "80b3a0d6-6486-4bff-a78d-c6ac5b265946"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:15:02.268605 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.268568    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:15:02.268605 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.268602    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knvfp\" (UniqueName: \"kubernetes.io/projected/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kube-api-access-knvfp\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:15:02.268605 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.268612    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80b3a0d6-6486-4bff-a78d-c6ac5b265946-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:15:02.268798 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.268621    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80b3a0d6-6486-4bff-a78d-c6ac5b265946-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:15:02.268798 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.268630    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80b3a0d6-6486-4bff-a78d-c6ac5b265946-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:15:02.350949 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.350924    2562 generic.go:356] "Generic (PLEG): container finished" podID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerID="ed43d2a2e996cdf4bb2aa7c504973385054fa3b2a0409a4b4ad740aa19303b35" exitCode=2
Jul 21 18:15:02.351108 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.350990    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" event={"ID":"7b60acdd-163f-48a3-a24e-c33cb88fe093","Type":"ContainerDied","Data":"ed43d2a2e996cdf4bb2aa7c504973385054fa3b2a0409a4b4ad740aa19303b35"}
Jul 21 18:15:02.352243 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.352223    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8_80b3a0d6-6486-4bff-a78d-c6ac5b265946/storage-initializer/1.log"
Jul 21 18:15:02.352353 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.352295    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8" event={"ID":"80b3a0d6-6486-4bff-a78d-c6ac5b265946","Type":"ContainerDied","Data":"bc399a2adb1c626bb004b146440fa481ec74571e7d0d60ac82d75c6b4e09d2b4"}
Jul 21 18:15:02.352353 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.352333    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"
Jul 21 18:15:02.352467 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.352354    2562 scope.go:117] "RemoveContainer" containerID="5e8bd71f7a4efae5091d2a8999dc95b23e77be9120f1a7d947a2e0c75012bbf7"
Jul 21 18:15:02.406885 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.406864    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"]
Jul 21 18:15:02.410817 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.410794    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-6e1ef9-predictor-6748bd7546-p2xk8"]
Jul 21 18:15:02.921407 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:02.921366    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80b3a0d6-6486-4bff-a78d-c6ac5b265946" path="/var/lib/kubelet/pods/80b3a0d6-6486-4bff-a78d-c6ac5b265946/volumes"
Jul 21 18:15:06.350477 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.350453    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:15:06.373374 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.373346    2562 generic.go:356] "Generic (PLEG): container finished" podID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerID="f9b2b4dbe36053e68c4b98b3e3a7a88f44f0b196a1ed5ae776c525d7f30e4bf4" exitCode=0
Jul 21 18:15:06.373520 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.373386    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" event={"ID":"7b60acdd-163f-48a3-a24e-c33cb88fe093","Type":"ContainerDied","Data":"f9b2b4dbe36053e68c4b98b3e3a7a88f44f0b196a1ed5ae776c525d7f30e4bf4"}
Jul 21 18:15:06.373520 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.373399    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"
Jul 21 18:15:06.373520 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.373446    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658" event={"ID":"7b60acdd-163f-48a3-a24e-c33cb88fe093","Type":"ContainerDied","Data":"b2b87498fb64ee05b925e63c6a64a6df2182488416d3a20b670839fad4da9496"}
Jul 21 18:15:06.373520 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.373487    2562 scope.go:117] "RemoveContainer" containerID="ed43d2a2e996cdf4bb2aa7c504973385054fa3b2a0409a4b4ad740aa19303b35"
Jul 21 18:15:06.381487 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.381467    2562 scope.go:117] "RemoveContainer" containerID="f9b2b4dbe36053e68c4b98b3e3a7a88f44f0b196a1ed5ae776c525d7f30e4bf4"
Jul 21 18:15:06.388677 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.388660    2562 scope.go:117] "RemoveContainer" containerID="aeff6080822faa6e08bcdee19c34f7423a345c0065c6484924a7839f395579a0"
Jul 21 18:15:06.395924 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.395909    2562 scope.go:117] "RemoveContainer" containerID="ed43d2a2e996cdf4bb2aa7c504973385054fa3b2a0409a4b4ad740aa19303b35"
Jul 21 18:15:06.396285 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:15:06.396263    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed43d2a2e996cdf4bb2aa7c504973385054fa3b2a0409a4b4ad740aa19303b35\": container with ID starting with ed43d2a2e996cdf4bb2aa7c504973385054fa3b2a0409a4b4ad740aa19303b35 not found: ID does not exist" containerID="ed43d2a2e996cdf4bb2aa7c504973385054fa3b2a0409a4b4ad740aa19303b35"
Jul 21 18:15:06.396355 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.396296    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed43d2a2e996cdf4bb2aa7c504973385054fa3b2a0409a4b4ad740aa19303b35"} err="failed to get container status \"ed43d2a2e996cdf4bb2aa7c504973385054fa3b2a0409a4b4ad740aa19303b35\": rpc error: code = NotFound desc = could not find container \"ed43d2a2e996cdf4bb2aa7c504973385054fa3b2a0409a4b4ad740aa19303b35\": container with ID starting with ed43d2a2e996cdf4bb2aa7c504973385054fa3b2a0409a4b4ad740aa19303b35 not found: ID does not exist"
Jul 21 18:15:06.396355 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.396315    2562 scope.go:117] "RemoveContainer" containerID="f9b2b4dbe36053e68c4b98b3e3a7a88f44f0b196a1ed5ae776c525d7f30e4bf4"
Jul 21 18:15:06.396609 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:15:06.396584    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b2b4dbe36053e68c4b98b3e3a7a88f44f0b196a1ed5ae776c525d7f30e4bf4\": container with ID starting with f9b2b4dbe36053e68c4b98b3e3a7a88f44f0b196a1ed5ae776c525d7f30e4bf4 not found: ID does not exist" containerID="f9b2b4dbe36053e68c4b98b3e3a7a88f44f0b196a1ed5ae776c525d7f30e4bf4"
Jul 21 18:15:06.396661 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.396619    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b2b4dbe36053e68c4b98b3e3a7a88f44f0b196a1ed5ae776c525d7f30e4bf4"} err="failed to get container status \"f9b2b4dbe36053e68c4b98b3e3a7a88f44f0b196a1ed5ae776c525d7f30e4bf4\": rpc error: code = NotFound desc = could not find container \"f9b2b4dbe36053e68c4b98b3e3a7a88f44f0b196a1ed5ae776c525d7f30e4bf4\": container with ID starting with f9b2b4dbe36053e68c4b98b3e3a7a88f44f0b196a1ed5ae776c525d7f30e4bf4 not found: ID does not exist"
Jul 21 18:15:06.396661 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.396642    2562 scope.go:117] "RemoveContainer" containerID="aeff6080822faa6e08bcdee19c34f7423a345c0065c6484924a7839f395579a0"
Jul 21 18:15:06.396860 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:15:06.396846    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeff6080822faa6e08bcdee19c34f7423a345c0065c6484924a7839f395579a0\": container with ID starting with aeff6080822faa6e08bcdee19c34f7423a345c0065c6484924a7839f395579a0 not found: ID does not exist" containerID="aeff6080822faa6e08bcdee19c34f7423a345c0065c6484924a7839f395579a0"
Jul 21 18:15:06.396904 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.396865    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeff6080822faa6e08bcdee19c34f7423a345c0065c6484924a7839f395579a0"} err="failed to get container status \"aeff6080822faa6e08bcdee19c34f7423a345c0065c6484924a7839f395579a0\": rpc error: code = NotFound desc = could not find container \"aeff6080822faa6e08bcdee19c34f7423a345c0065c6484924a7839f395579a0\": container with ID starting with aeff6080822faa6e08bcdee19c34f7423a345c0065c6484924a7839f395579a0 not found: ID does not exist"
Jul 21 18:15:06.499591 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.499517    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b60acdd-163f-48a3-a24e-c33cb88fe093-kube-rbac-proxy-sar-config\") pod \"7b60acdd-163f-48a3-a24e-c33cb88fe093\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") "
Jul 21 18:15:06.499591 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.499575    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b60acdd-163f-48a3-a24e-c33cb88fe093-kserve-provision-location\") pod \"7b60acdd-163f-48a3-a24e-c33cb88fe093\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") "
Jul 21 18:15:06.499783 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.499674    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkzfs\" (UniqueName: \"kubernetes.io/projected/7b60acdd-163f-48a3-a24e-c33cb88fe093-kube-api-access-bkzfs\") pod \"7b60acdd-163f-48a3-a24e-c33cb88fe093\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") "
Jul 21 18:15:06.499783 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.499752    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b60acdd-163f-48a3-a24e-c33cb88fe093-proxy-tls\") pod \"7b60acdd-163f-48a3-a24e-c33cb88fe093\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") "
Jul 21 18:15:06.499891 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.499806    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b60acdd-163f-48a3-a24e-c33cb88fe093-cabundle-cert\") pod \"7b60acdd-163f-48a3-a24e-c33cb88fe093\" (UID: \"7b60acdd-163f-48a3-a24e-c33cb88fe093\") "
Jul 21 18:15:06.499891 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.499861    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b60acdd-163f-48a3-a24e-c33cb88fe093-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b60acdd-163f-48a3-a24e-c33cb88fe093" (UID: "7b60acdd-163f-48a3-a24e-c33cb88fe093"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:15:06.499891 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.499862    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b60acdd-163f-48a3-a24e-c33cb88fe093-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b60acdd-163f-48a3-a24e-c33cb88fe093" (UID: "7b60acdd-163f-48a3-a24e-c33cb88fe093"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:15:06.500079 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.500048    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b60acdd-163f-48a3-a24e-c33cb88fe093-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:15:06.500145 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.500094    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b60acdd-163f-48a3-a24e-c33cb88fe093-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:15:06.500236 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.500211    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b60acdd-163f-48a3-a24e-c33cb88fe093-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7b60acdd-163f-48a3-a24e-c33cb88fe093" (UID: "7b60acdd-163f-48a3-a24e-c33cb88fe093"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:15:06.501867 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.501841    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b60acdd-163f-48a3-a24e-c33cb88fe093-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b60acdd-163f-48a3-a24e-c33cb88fe093" (UID: "7b60acdd-163f-48a3-a24e-c33cb88fe093"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:15:06.501867 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.501848    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b60acdd-163f-48a3-a24e-c33cb88fe093-kube-api-access-bkzfs" (OuterVolumeSpecName: "kube-api-access-bkzfs") pod "7b60acdd-163f-48a3-a24e-c33cb88fe093" (UID: "7b60acdd-163f-48a3-a24e-c33cb88fe093"). InnerVolumeSpecName "kube-api-access-bkzfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:15:06.600814 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.600776    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkzfs\" (UniqueName: \"kubernetes.io/projected/7b60acdd-163f-48a3-a24e-c33cb88fe093-kube-api-access-bkzfs\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:15:06.600814 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.600811    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b60acdd-163f-48a3-a24e-c33cb88fe093-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:15:06.600814 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.600821    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b60acdd-163f-48a3-a24e-c33cb88fe093-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:15:06.696511 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.696475    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"]
Jul 21 18:15:06.706143 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.706115    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-6e1ef9-predictor-5d6dbdb77-qr658"]
Jul 21 18:15:06.921355 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:06.921325    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" path="/var/lib/kubelet/pods/7b60acdd-163f-48a3-a24e-c33cb88fe093/volumes"
Jul 21 18:15:07.083032 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.083000    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"]
Jul 21 18:15:07.083332 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.083320    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="storage-initializer"
Jul 21 18:15:07.083378 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.083334    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="storage-initializer"
Jul 21 18:15:07.083378 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.083346    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b3a0d6-6486-4bff-a78d-c6ac5b265946" containerName="storage-initializer"
Jul 21 18:15:07.083378 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.083352    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b3a0d6-6486-4bff-a78d-c6ac5b265946" containerName="storage-initializer"
Jul 21 18:15:07.083378 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.083360    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kube-rbac-proxy"
Jul 21 18:15:07.083378 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.083367    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kube-rbac-proxy"
Jul 21 18:15:07.083378 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.083377    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kserve-container"
Jul 21 18:15:07.083378 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.083382    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kserve-container"
Jul 21 18:15:07.083609 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.083392    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b3a0d6-6486-4bff-a78d-c6ac5b265946" containerName="storage-initializer"
Jul 21 18:15:07.083609 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.083397    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b3a0d6-6486-4bff-a78d-c6ac5b265946" containerName="storage-initializer"
Jul 21 18:15:07.083609 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.083443    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b3a0d6-6486-4bff-a78d-c6ac5b265946" containerName="storage-initializer"
Jul 21 18:15:07.083609 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.083452    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b3a0d6-6486-4bff-a78d-c6ac5b265946" containerName="storage-initializer"
Jul 21 18:15:07.083609 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.083459    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kserve-container"
Jul 21 18:15:07.083609 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.083463    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b60acdd-163f-48a3-a24e-c33cb88fe093" containerName="kube-rbac-proxy"
Jul 21 18:15:07.087873 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.087856    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.090698 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.090675    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-f90a62-kube-rbac-proxy-sar-config\""
Jul 21 18:15:07.090808 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.090698    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:15:07.090808 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.090679    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-f90a62\""
Jul 21 18:15:07.090808 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.090798    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:15:07.090984 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.090970    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-f90a62-predictor-serving-cert\""
Jul 21 18:15:07.091049 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.091012    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-f90a62-dockercfg-45rqs\""
Jul 21 18:15:07.091862 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.091843    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:15:07.097566 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.097544    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"]
Jul 21 18:15:07.206214 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.206153    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsxvb\" (UniqueName: \"kubernetes.io/projected/11332f70-549d-4c11-a626-a032dfbf1365-kube-api-access-fsxvb\") pod \"isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") " pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.206214 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.206204    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11332f70-549d-4c11-a626-a032dfbf1365-kserve-provision-location\") pod \"isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") " pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.206372 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.206232    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11332f70-549d-4c11-a626-a032dfbf1365-cabundle-cert\") pod \"isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") " pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.206372 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.206296    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11332f70-549d-4c11-a626-a032dfbf1365-proxy-tls\") pod \"isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") " pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.206372 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.206318    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11332f70-549d-4c11-a626-a032dfbf1365-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") " pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.306750 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.306722    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11332f70-549d-4c11-a626-a032dfbf1365-kserve-provision-location\") pod \"isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") " pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.306881 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.306764    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11332f70-549d-4c11-a626-a032dfbf1365-cabundle-cert\") pod \"isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") " pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.306881 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.306821    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11332f70-549d-4c11-a626-a032dfbf1365-proxy-tls\") pod \"isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") " pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.306881 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.306857    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11332f70-549d-4c11-a626-a032dfbf1365-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") " pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.307044 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.306897    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsxvb\" (UniqueName: \"kubernetes.io/projected/11332f70-549d-4c11-a626-a032dfbf1365-kube-api-access-fsxvb\") pod \"isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") " pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.307151 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.307128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11332f70-549d-4c11-a626-a032dfbf1365-kserve-provision-location\") pod \"isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") " pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.307573 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.307524    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11332f70-549d-4c11-a626-a032dfbf1365-cabundle-cert\") pod \"isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") " pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.307573 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.307546    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11332f70-549d-4c11-a626-a032dfbf1365-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") " pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.309465 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.309444    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11332f70-549d-4c11-a626-a032dfbf1365-proxy-tls\") pod \"isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") " pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.315045 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.315025    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsxvb\" (UniqueName: \"kubernetes.io/projected/11332f70-549d-4c11-a626-a032dfbf1365-kube-api-access-fsxvb\") pod \"isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") " pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.398214 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.398185    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:07.529161 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:07.529137    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"]
Jul 21 18:15:07.531332 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:15:07.531308    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11332f70_549d_4c11_a626_a032dfbf1365.slice/crio-c63122355884ce59929ef6f2ef00795ec3f82c1f55c9e4fc2898554b351af010 WatchSource:0}: Error finding container c63122355884ce59929ef6f2ef00795ec3f82c1f55c9e4fc2898554b351af010: Status 404 returned error can't find the container with id c63122355884ce59929ef6f2ef00795ec3f82c1f55c9e4fc2898554b351af010
Jul 21 18:15:08.385171 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:08.385134    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg" event={"ID":"11332f70-549d-4c11-a626-a032dfbf1365","Type":"ContainerStarted","Data":"0b8c2412297b8fa31620e259a06669e916df398dc87fc92bcc8ca2f425938cfe"}
Jul 21 18:15:08.385327 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:08.385201    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg" event={"ID":"11332f70-549d-4c11-a626-a032dfbf1365","Type":"ContainerStarted","Data":"c63122355884ce59929ef6f2ef00795ec3f82c1f55c9e4fc2898554b351af010"}
Jul 21 18:15:11.400883 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:11.400855    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg_11332f70-549d-4c11-a626-a032dfbf1365/storage-initializer/0.log"
Jul 21 18:15:11.401225 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:11.400891    2562 generic.go:356] "Generic (PLEG): container finished" podID="11332f70-549d-4c11-a626-a032dfbf1365" containerID="0b8c2412297b8fa31620e259a06669e916df398dc87fc92bcc8ca2f425938cfe" exitCode=1
Jul 21 18:15:11.401225 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:11.400936    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg" event={"ID":"11332f70-549d-4c11-a626-a032dfbf1365","Type":"ContainerDied","Data":"0b8c2412297b8fa31620e259a06669e916df398dc87fc92bcc8ca2f425938cfe"}
Jul 21 18:15:12.076764 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:12.076727    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"]
Jul 21 18:15:12.407903 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:12.407877    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg_11332f70-549d-4c11-a626-a032dfbf1365/storage-initializer/0.log"
Jul 21 18:15:12.408308 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:12.407974    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg" event={"ID":"11332f70-549d-4c11-a626-a032dfbf1365","Type":"ContainerStarted","Data":"f01e7e14499a5353ee4caa68bf28611e26fd58923e6c40026bb1897e791b03fc"}
Jul 21 18:15:12.408308 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:12.408055    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg" podUID="11332f70-549d-4c11-a626-a032dfbf1365" containerName="storage-initializer" containerID="cri-o://f01e7e14499a5353ee4caa68bf28611e26fd58923e6c40026bb1897e791b03fc" gracePeriod=30
Jul 21 18:15:18.746186 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.746164    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg_11332f70-549d-4c11-a626-a032dfbf1365/storage-initializer/1.log"
Jul 21 18:15:18.746546 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.746534    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg_11332f70-549d-4c11-a626-a032dfbf1365/storage-initializer/0.log"
Jul 21 18:15:18.748769 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.748755    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:18.798301 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.798228    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11332f70-549d-4c11-a626-a032dfbf1365-cabundle-cert\") pod \"11332f70-549d-4c11-a626-a032dfbf1365\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") "
Jul 21 18:15:18.798301 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.798265    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11332f70-549d-4c11-a626-a032dfbf1365-kserve-provision-location\") pod \"11332f70-549d-4c11-a626-a032dfbf1365\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") "
Jul 21 18:15:18.798301 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.798295    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11332f70-549d-4c11-a626-a032dfbf1365-kube-rbac-proxy-sar-config\") pod \"11332f70-549d-4c11-a626-a032dfbf1365\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") "
Jul 21 18:15:18.798510 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.798317    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11332f70-549d-4c11-a626-a032dfbf1365-proxy-tls\") pod \"11332f70-549d-4c11-a626-a032dfbf1365\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") "
Jul 21 18:15:18.798510 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.798343    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsxvb\" (UniqueName: \"kubernetes.io/projected/11332f70-549d-4c11-a626-a032dfbf1365-kube-api-access-fsxvb\") pod \"11332f70-549d-4c11-a626-a032dfbf1365\" (UID: \"11332f70-549d-4c11-a626-a032dfbf1365\") "
Jul 21 18:15:18.798627 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.798505    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11332f70-549d-4c11-a626-a032dfbf1365-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "11332f70-549d-4c11-a626-a032dfbf1365" (UID: "11332f70-549d-4c11-a626-a032dfbf1365"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:15:18.798627 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.798607    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11332f70-549d-4c11-a626-a032dfbf1365-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "11332f70-549d-4c11-a626-a032dfbf1365" (UID: "11332f70-549d-4c11-a626-a032dfbf1365"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:15:18.798627 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.798612    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11332f70-549d-4c11-a626-a032dfbf1365-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "11332f70-549d-4c11-a626-a032dfbf1365" (UID: "11332f70-549d-4c11-a626-a032dfbf1365"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:15:18.800519 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.800481    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11332f70-549d-4c11-a626-a032dfbf1365-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "11332f70-549d-4c11-a626-a032dfbf1365" (UID: "11332f70-549d-4c11-a626-a032dfbf1365"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:15:18.800667 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.800540    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11332f70-549d-4c11-a626-a032dfbf1365-kube-api-access-fsxvb" (OuterVolumeSpecName: "kube-api-access-fsxvb") pod "11332f70-549d-4c11-a626-a032dfbf1365" (UID: "11332f70-549d-4c11-a626-a032dfbf1365"). InnerVolumeSpecName "kube-api-access-fsxvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:15:18.899379 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.899332    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11332f70-549d-4c11-a626-a032dfbf1365-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:15:18.899379 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.899374    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11332f70-549d-4c11-a626-a032dfbf1365-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:15:18.899379 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.899384    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11332f70-549d-4c11-a626-a032dfbf1365-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:15:18.899379 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.899394    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11332f70-549d-4c11-a626-a032dfbf1365-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:15:18.899624 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:18.899403    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsxvb\" (UniqueName: \"kubernetes.io/projected/11332f70-549d-4c11-a626-a032dfbf1365-kube-api-access-fsxvb\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:15:19.438696 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:19.438668    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg_11332f70-549d-4c11-a626-a032dfbf1365/storage-initializer/1.log"
Jul 21 18:15:19.439109 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:19.439094    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg_11332f70-549d-4c11-a626-a032dfbf1365/storage-initializer/0.log"
Jul 21 18:15:19.439184 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:19.439127    2562 generic.go:356] "Generic (PLEG): container finished" podID="11332f70-549d-4c11-a626-a032dfbf1365" containerID="f01e7e14499a5353ee4caa68bf28611e26fd58923e6c40026bb1897e791b03fc" exitCode=1
Jul 21 18:15:19.439225 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:19.439183    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"
Jul 21 18:15:19.439266 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:19.439184    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg" event={"ID":"11332f70-549d-4c11-a626-a032dfbf1365","Type":"ContainerDied","Data":"f01e7e14499a5353ee4caa68bf28611e26fd58923e6c40026bb1897e791b03fc"}
Jul 21 18:15:19.439319 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:19.439303    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg" event={"ID":"11332f70-549d-4c11-a626-a032dfbf1365","Type":"ContainerDied","Data":"c63122355884ce59929ef6f2ef00795ec3f82c1f55c9e4fc2898554b351af010"}
Jul 21 18:15:19.439354 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:19.439337    2562 scope.go:117] "RemoveContainer" containerID="f01e7e14499a5353ee4caa68bf28611e26fd58923e6c40026bb1897e791b03fc"
Jul 21 18:15:19.447689 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:19.447672    2562 scope.go:117] "RemoveContainer" containerID="0b8c2412297b8fa31620e259a06669e916df398dc87fc92bcc8ca2f425938cfe"
Jul 21 18:15:19.455212 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:19.455196    2562 scope.go:117] "RemoveContainer" containerID="f01e7e14499a5353ee4caa68bf28611e26fd58923e6c40026bb1897e791b03fc"
Jul 21 18:15:19.455486 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:15:19.455460    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01e7e14499a5353ee4caa68bf28611e26fd58923e6c40026bb1897e791b03fc\": container with ID starting with f01e7e14499a5353ee4caa68bf28611e26fd58923e6c40026bb1897e791b03fc not found: ID does not exist" containerID="f01e7e14499a5353ee4caa68bf28611e26fd58923e6c40026bb1897e791b03fc"
Jul 21 18:15:19.455583 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:19.455493    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01e7e14499a5353ee4caa68bf28611e26fd58923e6c40026bb1897e791b03fc"} err="failed to get container status \"f01e7e14499a5353ee4caa68bf28611e26fd58923e6c40026bb1897e791b03fc\": rpc error: code = NotFound desc = could not find container \"f01e7e14499a5353ee4caa68bf28611e26fd58923e6c40026bb1897e791b03fc\": container with ID starting with f01e7e14499a5353ee4caa68bf28611e26fd58923e6c40026bb1897e791b03fc not found: ID does not exist"
Jul 21 18:15:19.455583 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:19.455510    2562 scope.go:117] "RemoveContainer" containerID="0b8c2412297b8fa31620e259a06669e916df398dc87fc92bcc8ca2f425938cfe"
Jul 21 18:15:19.455707 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:15:19.455689    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8c2412297b8fa31620e259a06669e916df398dc87fc92bcc8ca2f425938cfe\": container with ID starting with 0b8c2412297b8fa31620e259a06669e916df398dc87fc92bcc8ca2f425938cfe not found: ID does not exist" containerID="0b8c2412297b8fa31620e259a06669e916df398dc87fc92bcc8ca2f425938cfe"
Jul 21 18:15:19.455745 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:19.455711    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8c2412297b8fa31620e259a06669e916df398dc87fc92bcc8ca2f425938cfe"} err="failed to get container status \"0b8c2412297b8fa31620e259a06669e916df398dc87fc92bcc8ca2f425938cfe\": rpc error: code = NotFound desc = could not find container \"0b8c2412297b8fa31620e259a06669e916df398dc87fc92bcc8ca2f425938cfe\": container with ID starting with 0b8c2412297b8fa31620e259a06669e916df398dc87fc92bcc8ca2f425938cfe not found: ID does not exist"
Jul 21 18:15:19.474580 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:19.474557    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"]
Jul 21 18:15:19.479222 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:19.479202    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f90a62-predictor-76fc84c75b-sf8vg"]
Jul 21 18:15:20.921558 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:20.921524    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11332f70-549d-4c11-a626-a032dfbf1365" path="/var/lib/kubelet/pods/11332f70-549d-4c11-a626-a032dfbf1365/volumes"
Jul 21 18:15:49.071083 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:49.071040    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:15:49.075122 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:15:49.075102    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:20:49.098392 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:20:49.098364    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:20:49.101268 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:20:49.101247    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:25:26.257764 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.257727    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"]
Jul 21 18:25:26.258393 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.258244    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11332f70-549d-4c11-a626-a032dfbf1365" containerName="storage-initializer"
Jul 21 18:25:26.258393 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.258264    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="11332f70-549d-4c11-a626-a032dfbf1365" containerName="storage-initializer"
Jul 21 18:25:26.258393 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.258281    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11332f70-549d-4c11-a626-a032dfbf1365" containerName="storage-initializer"
Jul 21 18:25:26.258393 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.258289    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="11332f70-549d-4c11-a626-a032dfbf1365" containerName="storage-initializer"
Jul 21 18:25:26.258393 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.258382    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="11332f70-549d-4c11-a626-a032dfbf1365" containerName="storage-initializer"
Jul 21 18:25:26.258393 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.258397    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="11332f70-549d-4c11-a626-a032dfbf1365" containerName="storage-initializer"
Jul 21 18:25:26.261662 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.261640    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.264271 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.264250    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:25:26.264383 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.264345    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:25:26.264730 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.264710    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 21 18:25:26.265430 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.265414    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 21 18:25:26.265499 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.265416    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:25:26.266009 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.265996    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:25:26.266055 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.266036    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:25:26.271165 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.271147    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"]
Jul 21 18:25:26.318143 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.318118    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c71655f6-735c-4662-86e3-cd1587e8b071-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-df6bcb64d-h5gqc\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.318259 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.318163    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c71655f6-735c-4662-86e3-cd1587e8b071-cabundle-cert\") pod \"isvc-sklearn-predictor-df6bcb64d-h5gqc\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.318259 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.318183    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c71655f6-735c-4662-86e3-cd1587e8b071-proxy-tls\") pod \"isvc-sklearn-predictor-df6bcb64d-h5gqc\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.318259 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.318236    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g28ww\" (UniqueName: \"kubernetes.io/projected/c71655f6-735c-4662-86e3-cd1587e8b071-kube-api-access-g28ww\") pod \"isvc-sklearn-predictor-df6bcb64d-h5gqc\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.318362 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.318295    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c71655f6-735c-4662-86e3-cd1587e8b071-kserve-provision-location\") pod \"isvc-sklearn-predictor-df6bcb64d-h5gqc\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.418712 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.418684    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c71655f6-735c-4662-86e3-cd1587e8b071-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-df6bcb64d-h5gqc\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.418868 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.418727    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c71655f6-735c-4662-86e3-cd1587e8b071-cabundle-cert\") pod \"isvc-sklearn-predictor-df6bcb64d-h5gqc\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.418868 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.418750    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c71655f6-735c-4662-86e3-cd1587e8b071-proxy-tls\") pod \"isvc-sklearn-predictor-df6bcb64d-h5gqc\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.418868 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.418770    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28ww\" (UniqueName: \"kubernetes.io/projected/c71655f6-735c-4662-86e3-cd1587e8b071-kube-api-access-g28ww\") pod \"isvc-sklearn-predictor-df6bcb64d-h5gqc\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.419042 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.418911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c71655f6-735c-4662-86e3-cd1587e8b071-kserve-provision-location\") pod \"isvc-sklearn-predictor-df6bcb64d-h5gqc\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.419299 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.419269    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c71655f6-735c-4662-86e3-cd1587e8b071-kserve-provision-location\") pod \"isvc-sklearn-predictor-df6bcb64d-h5gqc\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.419441 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.419422    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c71655f6-735c-4662-86e3-cd1587e8b071-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-df6bcb64d-h5gqc\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.419482 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.419439    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c71655f6-735c-4662-86e3-cd1587e8b071-cabundle-cert\") pod \"isvc-sklearn-predictor-df6bcb64d-h5gqc\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.421412 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.421394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c71655f6-735c-4662-86e3-cd1587e8b071-proxy-tls\") pod \"isvc-sklearn-predictor-df6bcb64d-h5gqc\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.427022 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.426999    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g28ww\" (UniqueName: \"kubernetes.io/projected/c71655f6-735c-4662-86e3-cd1587e8b071-kube-api-access-g28ww\") pod \"isvc-sklearn-predictor-df6bcb64d-h5gqc\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.572799 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.572710    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:26.710841 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.710817    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"]
Jul 21 18:25:26.713423 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:25:26.713388    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71655f6_735c_4662_86e3_cd1587e8b071.slice/crio-20659c2631fe35daec643559a970d0cb69cccb0019aaa9d666bddd903a6d46dc WatchSource:0}: Error finding container 20659c2631fe35daec643559a970d0cb69cccb0019aaa9d666bddd903a6d46dc: Status 404 returned error can't find the container with id 20659c2631fe35daec643559a970d0cb69cccb0019aaa9d666bddd903a6d46dc
Jul 21 18:25:26.819796 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.819766    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" event={"ID":"c71655f6-735c-4662-86e3-cd1587e8b071","Type":"ContainerStarted","Data":"adb03c5e76352521b4826dd629a467a1617a57889c66785eaec15272988a9d48"}
Jul 21 18:25:26.819924 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:26.819827    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" event={"ID":"c71655f6-735c-4662-86e3-cd1587e8b071","Type":"ContainerStarted","Data":"20659c2631fe35daec643559a970d0cb69cccb0019aaa9d666bddd903a6d46dc"}
Jul 21 18:25:30.842758 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:30.842669    2562 generic.go:356] "Generic (PLEG): container finished" podID="c71655f6-735c-4662-86e3-cd1587e8b071" containerID="adb03c5e76352521b4826dd629a467a1617a57889c66785eaec15272988a9d48" exitCode=0
Jul 21 18:25:30.842758 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:30.842711    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" event={"ID":"c71655f6-735c-4662-86e3-cd1587e8b071","Type":"ContainerDied","Data":"adb03c5e76352521b4826dd629a467a1617a57889c66785eaec15272988a9d48"}
Jul 21 18:25:31.848939 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:31.848901    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" event={"ID":"c71655f6-735c-4662-86e3-cd1587e8b071","Type":"ContainerStarted","Data":"868e529884145ae0278d54ff3b658b5dfb710fe8ea781ca7b6f49940b6d6c116"}
Jul 21 18:25:31.849508 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:31.848968    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" event={"ID":"c71655f6-735c-4662-86e3-cd1587e8b071","Type":"ContainerStarted","Data":"25d94d2c09a0faceed9af27b00c94777d8042e63cf655e143c877c5212b46d69"}
Jul 21 18:25:31.849508 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:31.849256    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:31.869267 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:31.869222    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" podStartSLOduration=2.062612897 podStartE2EDuration="5.869210033s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.806597136s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:25:31.867432019 +0000 UTC m=+3583.439053774" watchObservedRunningTime="2026-07-21 18:25:31.869210033 +0000 UTC m=+3583.440831788"
Jul 21 18:25:32.853762 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:32.853729    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:32.855125 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:32.855100    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 21 18:25:33.857858 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:33.857817    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 21 18:25:38.863020 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:38.862991    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:25:38.863667 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:38.863641    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 21 18:25:48.864240 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:48.864199    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 21 18:25:49.130706 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:49.130622    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:25:49.133897 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:49.133872    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:25:58.864298 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:25:58.864259    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 21 18:26:08.864436 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:26:08.864399    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 21 18:26:18.864312 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:26:18.864273    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 21 18:26:28.864389 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:26:28.864351    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 21 18:26:38.864531 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:26:38.864492    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 21 18:26:48.865039 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:26:48.865010    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:26:56.373010 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:26:56.372969    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"]
Jul 21 18:26:56.373519 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:26:56.373295    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kserve-container" containerID="cri-o://25d94d2c09a0faceed9af27b00c94777d8042e63cf655e143c877c5212b46d69" gracePeriod=30
Jul 21 18:26:56.373519 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:26:56.373295    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kube-rbac-proxy" containerID="cri-o://868e529884145ae0278d54ff3b658b5dfb710fe8ea781ca7b6f49940b6d6c116" gracePeriod=30
Jul 21 18:26:57.188964 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:26:57.188929    2562 generic.go:356] "Generic (PLEG): container finished" podID="c71655f6-735c-4662-86e3-cd1587e8b071" containerID="868e529884145ae0278d54ff3b658b5dfb710fe8ea781ca7b6f49940b6d6c116" exitCode=2
Jul 21 18:26:57.189152 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:26:57.188997    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" event={"ID":"c71655f6-735c-4662-86e3-cd1587e8b071","Type":"ContainerDied","Data":"868e529884145ae0278d54ff3b658b5dfb710fe8ea781ca7b6f49940b6d6c116"}
Jul 21 18:26:58.858388 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:26:58.858346    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Jul 21 18:26:58.863698 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:26:58.863674    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 21 18:27:00.817193 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:00.817171    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:27:00.983012 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:00.982968    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c71655f6-735c-4662-86e3-cd1587e8b071-cabundle-cert\") pod \"c71655f6-735c-4662-86e3-cd1587e8b071\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") "
Jul 21 18:27:00.983012 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:00.983015    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g28ww\" (UniqueName: \"kubernetes.io/projected/c71655f6-735c-4662-86e3-cd1587e8b071-kube-api-access-g28ww\") pod \"c71655f6-735c-4662-86e3-cd1587e8b071\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") "
Jul 21 18:27:00.983308 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:00.983045    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c71655f6-735c-4662-86e3-cd1587e8b071-kube-rbac-proxy-sar-config\") pod \"c71655f6-735c-4662-86e3-cd1587e8b071\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") "
Jul 21 18:27:00.983308 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:00.983084    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c71655f6-735c-4662-86e3-cd1587e8b071-proxy-tls\") pod \"c71655f6-735c-4662-86e3-cd1587e8b071\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") "
Jul 21 18:27:00.983308 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:00.983141    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c71655f6-735c-4662-86e3-cd1587e8b071-kserve-provision-location\") pod \"c71655f6-735c-4662-86e3-cd1587e8b071\" (UID: \"c71655f6-735c-4662-86e3-cd1587e8b071\") "
Jul 21 18:27:00.983498 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:00.983474    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71655f6-735c-4662-86e3-cd1587e8b071-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c71655f6-735c-4662-86e3-cd1587e8b071" (UID: "c71655f6-735c-4662-86e3-cd1587e8b071"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:27:00.983556 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:00.983400    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71655f6-735c-4662-86e3-cd1587e8b071-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c71655f6-735c-4662-86e3-cd1587e8b071" (UID: "c71655f6-735c-4662-86e3-cd1587e8b071"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:27:00.983556 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:00.983507    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71655f6-735c-4662-86e3-cd1587e8b071-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c71655f6-735c-4662-86e3-cd1587e8b071" (UID: "c71655f6-735c-4662-86e3-cd1587e8b071"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:27:00.985338 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:00.985311    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71655f6-735c-4662-86e3-cd1587e8b071-kube-api-access-g28ww" (OuterVolumeSpecName: "kube-api-access-g28ww") pod "c71655f6-735c-4662-86e3-cd1587e8b071" (UID: "c71655f6-735c-4662-86e3-cd1587e8b071"). InnerVolumeSpecName "kube-api-access-g28ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:27:00.985338 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:00.985308    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71655f6-735c-4662-86e3-cd1587e8b071-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c71655f6-735c-4662-86e3-cd1587e8b071" (UID: "c71655f6-735c-4662-86e3-cd1587e8b071"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:27:01.083925 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.083892    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c71655f6-735c-4662-86e3-cd1587e8b071-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:27:01.083925 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.083921    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c71655f6-735c-4662-86e3-cd1587e8b071-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:27:01.084191 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.083935    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g28ww\" (UniqueName: \"kubernetes.io/projected/c71655f6-735c-4662-86e3-cd1587e8b071-kube-api-access-g28ww\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:27:01.084191 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.083949    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c71655f6-735c-4662-86e3-cd1587e8b071-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:27:01.084191 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.083963    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c71655f6-735c-4662-86e3-cd1587e8b071-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:27:01.208250 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.208218    2562 generic.go:356] "Generic (PLEG): container finished" podID="c71655f6-735c-4662-86e3-cd1587e8b071" containerID="25d94d2c09a0faceed9af27b00c94777d8042e63cf655e143c877c5212b46d69" exitCode=0
Jul 21 18:27:01.208385 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.208277    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"
Jul 21 18:27:01.208385 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.208301    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" event={"ID":"c71655f6-735c-4662-86e3-cd1587e8b071","Type":"ContainerDied","Data":"25d94d2c09a0faceed9af27b00c94777d8042e63cf655e143c877c5212b46d69"}
Jul 21 18:27:01.208385 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.208360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc" event={"ID":"c71655f6-735c-4662-86e3-cd1587e8b071","Type":"ContainerDied","Data":"20659c2631fe35daec643559a970d0cb69cccb0019aaa9d666bddd903a6d46dc"}
Jul 21 18:27:01.208506 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.208390    2562 scope.go:117] "RemoveContainer" containerID="868e529884145ae0278d54ff3b658b5dfb710fe8ea781ca7b6f49940b6d6c116"
Jul 21 18:27:01.216913 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.216898    2562 scope.go:117] "RemoveContainer" containerID="25d94d2c09a0faceed9af27b00c94777d8042e63cf655e143c877c5212b46d69"
Jul 21 18:27:01.224392 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.224377    2562 scope.go:117] "RemoveContainer" containerID="adb03c5e76352521b4826dd629a467a1617a57889c66785eaec15272988a9d48"
Jul 21 18:27:01.231113 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.231092    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"]
Jul 21 18:27:01.231796 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.231782    2562 scope.go:117] "RemoveContainer" containerID="868e529884145ae0278d54ff3b658b5dfb710fe8ea781ca7b6f49940b6d6c116"
Jul 21 18:27:01.232036 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:27:01.232016    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868e529884145ae0278d54ff3b658b5dfb710fe8ea781ca7b6f49940b6d6c116\": container with ID starting with 868e529884145ae0278d54ff3b658b5dfb710fe8ea781ca7b6f49940b6d6c116 not found: ID does not exist" containerID="868e529884145ae0278d54ff3b658b5dfb710fe8ea781ca7b6f49940b6d6c116"
Jul 21 18:27:01.232094 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.232045    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868e529884145ae0278d54ff3b658b5dfb710fe8ea781ca7b6f49940b6d6c116"} err="failed to get container status \"868e529884145ae0278d54ff3b658b5dfb710fe8ea781ca7b6f49940b6d6c116\": rpc error: code = NotFound desc = could not find container \"868e529884145ae0278d54ff3b658b5dfb710fe8ea781ca7b6f49940b6d6c116\": container with ID starting with 868e529884145ae0278d54ff3b658b5dfb710fe8ea781ca7b6f49940b6d6c116 not found: ID does not exist"
Jul 21 18:27:01.232094 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.232081    2562 scope.go:117] "RemoveContainer" containerID="25d94d2c09a0faceed9af27b00c94777d8042e63cf655e143c877c5212b46d69"
Jul 21 18:27:01.232285 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:27:01.232269    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d94d2c09a0faceed9af27b00c94777d8042e63cf655e143c877c5212b46d69\": container with ID starting with 25d94d2c09a0faceed9af27b00c94777d8042e63cf655e143c877c5212b46d69 not found: ID does not exist" containerID="25d94d2c09a0faceed9af27b00c94777d8042e63cf655e143c877c5212b46d69"
Jul 21 18:27:01.232322 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.232291    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d94d2c09a0faceed9af27b00c94777d8042e63cf655e143c877c5212b46d69"} err="failed to get container status \"25d94d2c09a0faceed9af27b00c94777d8042e63cf655e143c877c5212b46d69\": rpc error: code = NotFound desc = could not find container \"25d94d2c09a0faceed9af27b00c94777d8042e63cf655e143c877c5212b46d69\": container with ID starting with 25d94d2c09a0faceed9af27b00c94777d8042e63cf655e143c877c5212b46d69 not found: ID does not exist"
Jul 21 18:27:01.232322 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.232307    2562 scope.go:117] "RemoveContainer" containerID="adb03c5e76352521b4826dd629a467a1617a57889c66785eaec15272988a9d48"
Jul 21 18:27:01.232508 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:27:01.232492    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb03c5e76352521b4826dd629a467a1617a57889c66785eaec15272988a9d48\": container with ID starting with adb03c5e76352521b4826dd629a467a1617a57889c66785eaec15272988a9d48 not found: ID does not exist" containerID="adb03c5e76352521b4826dd629a467a1617a57889c66785eaec15272988a9d48"
Jul 21 18:27:01.232548 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.232513    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb03c5e76352521b4826dd629a467a1617a57889c66785eaec15272988a9d48"} err="failed to get container status \"adb03c5e76352521b4826dd629a467a1617a57889c66785eaec15272988a9d48\": rpc error: code = NotFound desc = could not find container \"adb03c5e76352521b4826dd629a467a1617a57889c66785eaec15272988a9d48\": container with ID starting with adb03c5e76352521b4826dd629a467a1617a57889c66785eaec15272988a9d48 not found: ID does not exist"
Jul 21 18:27:01.234938 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.234894    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-df6bcb64d-h5gqc"]
Jul 21 18:27:01.561352 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.561111    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"]
Jul 21 18:27:01.561552 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.561535    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="storage-initializer"
Jul 21 18:27:01.561592 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.561554    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="storage-initializer"
Jul 21 18:27:01.561592 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.561568    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kserve-container"
Jul 21 18:27:01.561592 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.561574    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kserve-container"
Jul 21 18:27:01.561592 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.561586    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kube-rbac-proxy"
Jul 21 18:27:01.561592 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.561593    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kube-rbac-proxy"
Jul 21 18:27:01.561749 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.561647    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kube-rbac-proxy"
Jul 21 18:27:01.561749 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.561655    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" containerName="kserve-container"
Jul 21 18:27:01.565984 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.565965    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:01.569056 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.569028    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:27:01.570060 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.569981    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:27:01.570060 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.570006    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:27:01.570060 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.570025    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:27:01.570060 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.570026    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 21 18:27:01.570342 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.570113    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:27:01.570342 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.570026    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 21 18:27:01.576125 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.576105    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"]
Jul 21 18:27:01.688423 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.688383    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6765d34-ec56-4bfe-acd7-ca86865af464-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:01.688608 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.688476    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6765d34-ec56-4bfe-acd7-ca86865af464-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:01.688608 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.688516    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6765d34-ec56-4bfe-acd7-ca86865af464-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:01.688608 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.688544    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6765d34-ec56-4bfe-acd7-ca86865af464-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:01.688608 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.688592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lsfm\" (UniqueName: \"kubernetes.io/projected/d6765d34-ec56-4bfe-acd7-ca86865af464-kube-api-access-2lsfm\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:01.790005 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.789962    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lsfm\" (UniqueName: \"kubernetes.io/projected/d6765d34-ec56-4bfe-acd7-ca86865af464-kube-api-access-2lsfm\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:01.790220 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.790106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6765d34-ec56-4bfe-acd7-ca86865af464-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:01.790220 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.790153    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6765d34-ec56-4bfe-acd7-ca86865af464-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:01.790220 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.790169    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6765d34-ec56-4bfe-acd7-ca86865af464-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:01.790220 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.790194    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6765d34-ec56-4bfe-acd7-ca86865af464-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:01.790587 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.790548    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6765d34-ec56-4bfe-acd7-ca86865af464-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:01.790872 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.790852    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6765d34-ec56-4bfe-acd7-ca86865af464-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:01.790956 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.790884    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6765d34-ec56-4bfe-acd7-ca86865af464-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:01.792923 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.792900    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6765d34-ec56-4bfe-acd7-ca86865af464-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:01.798653 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.798632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lsfm\" (UniqueName: \"kubernetes.io/projected/d6765d34-ec56-4bfe-acd7-ca86865af464-kube-api-access-2lsfm\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:01.876832 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:01.876747    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:02.007802 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:02.007718    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"]
Jul 21 18:27:02.009981 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:27:02.009955    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6765d34_ec56_4bfe_acd7_ca86865af464.slice/crio-7751977852c78164d560fcf82299dd5bfe7e6da30bdc82ee62d7110a7f519604 WatchSource:0}: Error finding container 7751977852c78164d560fcf82299dd5bfe7e6da30bdc82ee62d7110a7f519604: Status 404 returned error can't find the container with id 7751977852c78164d560fcf82299dd5bfe7e6da30bdc82ee62d7110a7f519604
Jul 21 18:27:02.214626 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:02.214588    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm" event={"ID":"d6765d34-ec56-4bfe-acd7-ca86865af464","Type":"ContainerStarted","Data":"058022a6cb24769e6e48f44c760890f7c9b6a965fabc8a4ed49c4b269cf06a7f"}
Jul 21 18:27:02.214827 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:02.214657    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm" event={"ID":"d6765d34-ec56-4bfe-acd7-ca86865af464","Type":"ContainerStarted","Data":"7751977852c78164d560fcf82299dd5bfe7e6da30bdc82ee62d7110a7f519604"}
Jul 21 18:27:02.921568 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:02.921535    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c71655f6-735c-4662-86e3-cd1587e8b071" path="/var/lib/kubelet/pods/c71655f6-735c-4662-86e3-cd1587e8b071/volumes"
Jul 21 18:27:06.234315 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:06.234282    2562 generic.go:356] "Generic (PLEG): container finished" podID="d6765d34-ec56-4bfe-acd7-ca86865af464" containerID="058022a6cb24769e6e48f44c760890f7c9b6a965fabc8a4ed49c4b269cf06a7f" exitCode=0
Jul 21 18:27:06.234816 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:06.234364    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm" event={"ID":"d6765d34-ec56-4bfe-acd7-ca86865af464","Type":"ContainerDied","Data":"058022a6cb24769e6e48f44c760890f7c9b6a965fabc8a4ed49c4b269cf06a7f"}
Jul 21 18:27:07.240389 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:07.240356    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm" event={"ID":"d6765d34-ec56-4bfe-acd7-ca86865af464","Type":"ContainerStarted","Data":"e3114ee58da3d31130fd735914878feed341fac4e0b3201f64384c012ebfba68"}
Jul 21 18:27:07.240869 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:07.240415    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm" event={"ID":"d6765d34-ec56-4bfe-acd7-ca86865af464","Type":"ContainerStarted","Data":"1174930ed737b3ea4294ba8df2622538ec478a510ac376a97e0e80bd1a5a9cde"}
Jul 21 18:27:07.240869 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:07.240611    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:07.262898 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:07.262845    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm" podStartSLOduration=2.751544578 podStartE2EDuration="6.262831169s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.511286591s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:27:07.260557896 +0000 UTC m=+3678.832179673" watchObservedRunningTime="2026-07-21 18:27:07.262831169 +0000 UTC m=+3678.834452923"
Jul 21 18:27:08.244571 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:08.244544    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:14.254047 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:14.254019    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:27:44.293435 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:44.293394    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm" podUID="d6765d34-ec56-4bfe-acd7-ca86865af464" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 21 18:27:54.256949 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:27:54.256918    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:28:01.618199 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:01.618162    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"]
Jul 21 18:28:01.618693 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:01.618470    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm" podUID="d6765d34-ec56-4bfe-acd7-ca86865af464" containerName="kserve-container" containerID="cri-o://1174930ed737b3ea4294ba8df2622538ec478a510ac376a97e0e80bd1a5a9cde" gracePeriod=30
Jul 21 18:28:01.618789 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:01.618754    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm" podUID="d6765d34-ec56-4bfe-acd7-ca86865af464" containerName="kube-rbac-proxy" containerID="cri-o://e3114ee58da3d31130fd735914878feed341fac4e0b3201f64384c012ebfba68" gracePeriod=30
Jul 21 18:28:02.473416 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:02.473381    2562 generic.go:356] "Generic (PLEG): container finished" podID="d6765d34-ec56-4bfe-acd7-ca86865af464" containerID="e3114ee58da3d31130fd735914878feed341fac4e0b3201f64384c012ebfba68" exitCode=2
Jul 21 18:28:02.473585 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:02.473432    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm" event={"ID":"d6765d34-ec56-4bfe-acd7-ca86865af464","Type":"ContainerDied","Data":"e3114ee58da3d31130fd735914878feed341fac4e0b3201f64384c012ebfba68"}
Jul 21 18:28:04.248788 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:04.248744    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm" podUID="d6765d34-ec56-4bfe-acd7-ca86865af464" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Jul 21 18:28:06.792577 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.792544    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"]
Jul 21 18:28:06.801331 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.801308    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:06.804123 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.804102    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jul 21 18:28:06.804123 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.804110    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jul 21 18:28:06.809606 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.809584    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"]
Jul 21 18:28:06.893650 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.893609    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e977334-378c-4cfa-a69a-b50729246cb3-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-757d748996-njdv7\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:06.893812 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.893657    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9xx4\" (UniqueName: \"kubernetes.io/projected/6e977334-378c-4cfa-a69a-b50729246cb3-kube-api-access-w9xx4\") pod \"isvc-sklearn-runtime-predictor-757d748996-njdv7\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:06.893812 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.893734    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e977334-378c-4cfa-a69a-b50729246cb3-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-757d748996-njdv7\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:06.893812 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.893760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e977334-378c-4cfa-a69a-b50729246cb3-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-757d748996-njdv7\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:06.893812 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.893787    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e977334-378c-4cfa-a69a-b50729246cb3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-757d748996-njdv7\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:06.994360 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.994326    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e977334-378c-4cfa-a69a-b50729246cb3-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-757d748996-njdv7\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:06.994360 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.994360    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9xx4\" (UniqueName: \"kubernetes.io/projected/6e977334-378c-4cfa-a69a-b50729246cb3-kube-api-access-w9xx4\") pod \"isvc-sklearn-runtime-predictor-757d748996-njdv7\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:06.994618 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.994386    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e977334-378c-4cfa-a69a-b50729246cb3-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-757d748996-njdv7\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:06.994618 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.994402    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e977334-378c-4cfa-a69a-b50729246cb3-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-757d748996-njdv7\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:06.994618 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.994451    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e977334-378c-4cfa-a69a-b50729246cb3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-757d748996-njdv7\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:06.994618 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:28:06.994498    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Jul 21 18:28:06.994618 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:28:06.994570    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e977334-378c-4cfa-a69a-b50729246cb3-proxy-tls podName:6e977334-378c-4cfa-a69a-b50729246cb3 nodeName:}" failed. No retries permitted until 2026-07-21 18:28:07.494547915 +0000 UTC m=+3739.066169653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6e977334-378c-4cfa-a69a-b50729246cb3-proxy-tls") pod "isvc-sklearn-runtime-predictor-757d748996-njdv7" (UID: "6e977334-378c-4cfa-a69a-b50729246cb3") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Jul 21 18:28:06.994839 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.994823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e977334-378c-4cfa-a69a-b50729246cb3-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-757d748996-njdv7\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:06.995051 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.995035    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e977334-378c-4cfa-a69a-b50729246cb3-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-757d748996-njdv7\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:06.995132 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:06.995115    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e977334-378c-4cfa-a69a-b50729246cb3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-757d748996-njdv7\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:07.003412 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:07.003395    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9xx4\" (UniqueName: \"kubernetes.io/projected/6e977334-378c-4cfa-a69a-b50729246cb3-kube-api-access-w9xx4\") pod \"isvc-sklearn-runtime-predictor-757d748996-njdv7\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:07.498951 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:07.498923    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e977334-378c-4cfa-a69a-b50729246cb3-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-757d748996-njdv7\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:07.501442 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:07.501413    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e977334-378c-4cfa-a69a-b50729246cb3-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-757d748996-njdv7\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:07.712701 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:07.712665    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:07.852709 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:07.852681    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"]
Jul 21 18:28:07.854972 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:28:07.854944    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e977334_378c_4cfa_a69a_b50729246cb3.slice/crio-ec4d542101c317219acfe57dff9b5719cf199d6bc27167396a616f6d9544790b WatchSource:0}: Error finding container ec4d542101c317219acfe57dff9b5719cf199d6bc27167396a616f6d9544790b: Status 404 returned error can't find the container with id ec4d542101c317219acfe57dff9b5719cf199d6bc27167396a616f6d9544790b
Jul 21 18:28:08.503582 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.503550    2562 generic.go:356] "Generic (PLEG): container finished" podID="d6765d34-ec56-4bfe-acd7-ca86865af464" containerID="1174930ed737b3ea4294ba8df2622538ec478a510ac376a97e0e80bd1a5a9cde" exitCode=0
Jul 21 18:28:08.503719 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.503623    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm" event={"ID":"d6765d34-ec56-4bfe-acd7-ca86865af464","Type":"ContainerDied","Data":"1174930ed737b3ea4294ba8df2622538ec478a510ac376a97e0e80bd1a5a9cde"}
Jul 21 18:28:08.505473 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.505431    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7" event={"ID":"6e977334-378c-4cfa-a69a-b50729246cb3","Type":"ContainerStarted","Data":"5b2545c086ab63ea412234a5ebfe215e0cfc1dc638724def25cf5bf93cf6ee9d"}
Jul 21 18:28:08.505591 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.505490    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7" event={"ID":"6e977334-378c-4cfa-a69a-b50729246cb3","Type":"ContainerStarted","Data":"ec4d542101c317219acfe57dff9b5719cf199d6bc27167396a616f6d9544790b"}
Jul 21 18:28:08.562095 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.562050    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:28:08.709266 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.709237    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6765d34-ec56-4bfe-acd7-ca86865af464-kserve-provision-location\") pod \"d6765d34-ec56-4bfe-acd7-ca86865af464\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") "
Jul 21 18:28:08.709266 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.709268    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6765d34-ec56-4bfe-acd7-ca86865af464-cabundle-cert\") pod \"d6765d34-ec56-4bfe-acd7-ca86865af464\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") "
Jul 21 18:28:08.709487 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.709313    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6765d34-ec56-4bfe-acd7-ca86865af464-kube-rbac-proxy-sar-config\") pod \"d6765d34-ec56-4bfe-acd7-ca86865af464\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") "
Jul 21 18:28:08.709487 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.709335    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6765d34-ec56-4bfe-acd7-ca86865af464-proxy-tls\") pod \"d6765d34-ec56-4bfe-acd7-ca86865af464\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") "
Jul 21 18:28:08.709487 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.709373    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lsfm\" (UniqueName: \"kubernetes.io/projected/d6765d34-ec56-4bfe-acd7-ca86865af464-kube-api-access-2lsfm\") pod \"d6765d34-ec56-4bfe-acd7-ca86865af464\" (UID: \"d6765d34-ec56-4bfe-acd7-ca86865af464\") "
Jul 21 18:28:08.709659 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.709571    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6765d34-ec56-4bfe-acd7-ca86865af464-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6765d34-ec56-4bfe-acd7-ca86865af464" (UID: "d6765d34-ec56-4bfe-acd7-ca86865af464"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:28:08.709659 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.709656    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6765d34-ec56-4bfe-acd7-ca86865af464-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d6765d34-ec56-4bfe-acd7-ca86865af464" (UID: "d6765d34-ec56-4bfe-acd7-ca86865af464"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:28:08.709755 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.709690    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6765d34-ec56-4bfe-acd7-ca86865af464-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d6765d34-ec56-4bfe-acd7-ca86865af464" (UID: "d6765d34-ec56-4bfe-acd7-ca86865af464"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:28:08.711546 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.711528    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6765d34-ec56-4bfe-acd7-ca86865af464-kube-api-access-2lsfm" (OuterVolumeSpecName: "kube-api-access-2lsfm") pod "d6765d34-ec56-4bfe-acd7-ca86865af464" (UID: "d6765d34-ec56-4bfe-acd7-ca86865af464"). InnerVolumeSpecName "kube-api-access-2lsfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:28:08.711729 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.711708    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6765d34-ec56-4bfe-acd7-ca86865af464-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6765d34-ec56-4bfe-acd7-ca86865af464" (UID: "d6765d34-ec56-4bfe-acd7-ca86865af464"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:28:08.810391 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.810356    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6765d34-ec56-4bfe-acd7-ca86865af464-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:28:08.810391 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.810383    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6765d34-ec56-4bfe-acd7-ca86865af464-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:28:08.810391 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.810396    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lsfm\" (UniqueName: \"kubernetes.io/projected/d6765d34-ec56-4bfe-acd7-ca86865af464-kube-api-access-2lsfm\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:28:08.810606 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.810408    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6765d34-ec56-4bfe-acd7-ca86865af464-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:28:08.810606 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:08.810421    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6765d34-ec56-4bfe-acd7-ca86865af464-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:28:09.512116 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:09.512086    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"
Jul 21 18:28:09.512116 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:09.512098    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm" event={"ID":"d6765d34-ec56-4bfe-acd7-ca86865af464","Type":"ContainerDied","Data":"7751977852c78164d560fcf82299dd5bfe7e6da30bdc82ee62d7110a7f519604"}
Jul 21 18:28:09.512615 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:09.512164    2562 scope.go:117] "RemoveContainer" containerID="e3114ee58da3d31130fd735914878feed341fac4e0b3201f64384c012ebfba68"
Jul 21 18:28:09.520750 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:09.520734    2562 scope.go:117] "RemoveContainer" containerID="1174930ed737b3ea4294ba8df2622538ec478a510ac376a97e0e80bd1a5a9cde"
Jul 21 18:28:09.527900 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:09.527880    2562 scope.go:117] "RemoveContainer" containerID="058022a6cb24769e6e48f44c760890f7c9b6a965fabc8a4ed49c4b269cf06a7f"
Jul 21 18:28:09.531890 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:09.531868    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"]
Jul 21 18:28:09.536683 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:09.536663    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bpcm"]
Jul 21 18:28:10.921153 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:10.921120    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6765d34-ec56-4bfe-acd7-ca86865af464" path="/var/lib/kubelet/pods/d6765d34-ec56-4bfe-acd7-ca86865af464/volumes"
Jul 21 18:28:13.531095 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:13.531046    2562 generic.go:356] "Generic (PLEG): container finished" podID="6e977334-378c-4cfa-a69a-b50729246cb3" containerID="5b2545c086ab63ea412234a5ebfe215e0cfc1dc638724def25cf5bf93cf6ee9d" exitCode=0
Jul 21 18:28:13.531485 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:13.531117    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7" event={"ID":"6e977334-378c-4cfa-a69a-b50729246cb3","Type":"ContainerDied","Data":"5b2545c086ab63ea412234a5ebfe215e0cfc1dc638724def25cf5bf93cf6ee9d"}
Jul 21 18:28:14.537696 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:14.537660    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7" event={"ID":"6e977334-378c-4cfa-a69a-b50729246cb3","Type":"ContainerStarted","Data":"f2dc35135c9d23e6f1dd53e86df6068791fb42f58f642bec18b941d5feab10d8"}
Jul 21 18:28:14.538156 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:14.537743    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7" event={"ID":"6e977334-378c-4cfa-a69a-b50729246cb3","Type":"ContainerStarted","Data":"fbcef7cd332113ddcd24cc27e949be370a093bc211e5d52d1ac5991f1e08a6fa"}
Jul 21 18:28:14.538156 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:14.537970    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:14.557620 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:14.557576    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7" podStartSLOduration=3.136811699 podStartE2EDuration="8.55756172s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.420750021s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:28:14.556868755 +0000 UTC m=+3746.128490558" watchObservedRunningTime="2026-07-21 18:28:14.55756172 +0000 UTC m=+3746.129183475"
Jul 21 18:28:15.542180 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:15.542145    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:15.543271 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:15.543245    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7" podUID="6e977334-378c-4cfa-a69a-b50729246cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 21 18:28:16.547623 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:16.547583    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7" podUID="6e977334-378c-4cfa-a69a-b50729246cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 21 18:28:21.553850 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:21.553817    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:21.554383 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:21.554292    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7" podUID="6e977334-378c-4cfa-a69a-b50729246cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 21 18:28:31.555262 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:31.555235    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:43.628855 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:43.628826    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-757d748996-njdv7_6e977334-378c-4cfa-a69a-b50729246cb3/kserve-container/0.log"
Jul 21 18:28:43.791532 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:43.791501    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"]
Jul 21 18:28:43.791799 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:43.791774    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7" podUID="6e977334-378c-4cfa-a69a-b50729246cb3" containerName="kserve-container" containerID="cri-o://fbcef7cd332113ddcd24cc27e949be370a093bc211e5d52d1ac5991f1e08a6fa" gracePeriod=30
Jul 21 18:28:43.791891 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:43.791804    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7" podUID="6e977334-378c-4cfa-a69a-b50729246cb3" containerName="kube-rbac-proxy" containerID="cri-o://f2dc35135c9d23e6f1dd53e86df6068791fb42f58f642bec18b941d5feab10d8" gracePeriod=30
Jul 21 18:28:44.682371 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:44.682340    2562 generic.go:356] "Generic (PLEG): container finished" podID="6e977334-378c-4cfa-a69a-b50729246cb3" containerID="f2dc35135c9d23e6f1dd53e86df6068791fb42f58f642bec18b941d5feab10d8" exitCode=2
Jul 21 18:28:44.682774 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:44.682396    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7" event={"ID":"6e977334-378c-4cfa-a69a-b50729246cb3","Type":"ContainerDied","Data":"f2dc35135c9d23e6f1dd53e86df6068791fb42f58f642bec18b941d5feab10d8"}
Jul 21 18:28:44.937748 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:44.937686    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:45.084244 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.084204    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e977334-378c-4cfa-a69a-b50729246cb3-proxy-tls\") pod \"6e977334-378c-4cfa-a69a-b50729246cb3\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") "
Jul 21 18:28:45.084413 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.084281    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e977334-378c-4cfa-a69a-b50729246cb3-kserve-provision-location\") pod \"6e977334-378c-4cfa-a69a-b50729246cb3\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") "
Jul 21 18:28:45.084463 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.084440    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e977334-378c-4cfa-a69a-b50729246cb3-cabundle-cert\") pod \"6e977334-378c-4cfa-a69a-b50729246cb3\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") "
Jul 21 18:28:45.084571 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.084543    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9xx4\" (UniqueName: \"kubernetes.io/projected/6e977334-378c-4cfa-a69a-b50729246cb3-kube-api-access-w9xx4\") pod \"6e977334-378c-4cfa-a69a-b50729246cb3\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") "
Jul 21 18:28:45.084722 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.084578    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e977334-378c-4cfa-a69a-b50729246cb3-kube-rbac-proxy-sar-config\") pod \"6e977334-378c-4cfa-a69a-b50729246cb3\" (UID: \"6e977334-378c-4cfa-a69a-b50729246cb3\") "
Jul 21 18:28:45.084848 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.084826    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e977334-378c-4cfa-a69a-b50729246cb3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e977334-378c-4cfa-a69a-b50729246cb3" (UID: "6e977334-378c-4cfa-a69a-b50729246cb3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:28:45.085101 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.084936    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e977334-378c-4cfa-a69a-b50729246cb3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e977334-378c-4cfa-a69a-b50729246cb3" (UID: "6e977334-378c-4cfa-a69a-b50729246cb3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:28:45.086459 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.086433    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e977334-378c-4cfa-a69a-b50729246cb3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e977334-378c-4cfa-a69a-b50729246cb3" (UID: "6e977334-378c-4cfa-a69a-b50729246cb3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:28:45.086659 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.086638    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e977334-378c-4cfa-a69a-b50729246cb3-kube-api-access-w9xx4" (OuterVolumeSpecName: "kube-api-access-w9xx4") pod "6e977334-378c-4cfa-a69a-b50729246cb3" (UID: "6e977334-378c-4cfa-a69a-b50729246cb3"). InnerVolumeSpecName "kube-api-access-w9xx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:28:45.110360 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.110334    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e977334-378c-4cfa-a69a-b50729246cb3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e977334-378c-4cfa-a69a-b50729246cb3" (UID: "6e977334-378c-4cfa-a69a-b50729246cb3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:28:45.186146 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.186121    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9xx4\" (UniqueName: \"kubernetes.io/projected/6e977334-378c-4cfa-a69a-b50729246cb3-kube-api-access-w9xx4\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:28:45.186260 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.186149    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e977334-378c-4cfa-a69a-b50729246cb3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:28:45.186260 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.186166    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e977334-378c-4cfa-a69a-b50729246cb3-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:28:45.186260 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.186183    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e977334-378c-4cfa-a69a-b50729246cb3-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:28:45.186260 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.186198    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e977334-378c-4cfa-a69a-b50729246cb3-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:28:45.694032 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.693988    2562 generic.go:356] "Generic (PLEG): container finished" podID="6e977334-378c-4cfa-a69a-b50729246cb3" containerID="fbcef7cd332113ddcd24cc27e949be370a093bc211e5d52d1ac5991f1e08a6fa" exitCode=0
Jul 21 18:28:45.694495 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.694167    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7" event={"ID":"6e977334-378c-4cfa-a69a-b50729246cb3","Type":"ContainerDied","Data":"fbcef7cd332113ddcd24cc27e949be370a093bc211e5d52d1ac5991f1e08a6fa"}
Jul 21 18:28:45.694495 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.694202    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"
Jul 21 18:28:45.694495 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.694257    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7" event={"ID":"6e977334-378c-4cfa-a69a-b50729246cb3","Type":"ContainerDied","Data":"ec4d542101c317219acfe57dff9b5719cf199d6bc27167396a616f6d9544790b"}
Jul 21 18:28:45.694495 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.694292    2562 scope.go:117] "RemoveContainer" containerID="f2dc35135c9d23e6f1dd53e86df6068791fb42f58f642bec18b941d5feab10d8"
Jul 21 18:28:45.703435 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.703413    2562 scope.go:117] "RemoveContainer" containerID="fbcef7cd332113ddcd24cc27e949be370a093bc211e5d52d1ac5991f1e08a6fa"
Jul 21 18:28:45.710878 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.710862    2562 scope.go:117] "RemoveContainer" containerID="5b2545c086ab63ea412234a5ebfe215e0cfc1dc638724def25cf5bf93cf6ee9d"
Jul 21 18:28:45.715781 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.715761    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"]
Jul 21 18:28:45.718602 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.718553    2562 scope.go:117] "RemoveContainer" containerID="f2dc35135c9d23e6f1dd53e86df6068791fb42f58f642bec18b941d5feab10d8"
Jul 21 18:28:45.718827 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:28:45.718808    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2dc35135c9d23e6f1dd53e86df6068791fb42f58f642bec18b941d5feab10d8\": container with ID starting with f2dc35135c9d23e6f1dd53e86df6068791fb42f58f642bec18b941d5feab10d8 not found: ID does not exist" containerID="f2dc35135c9d23e6f1dd53e86df6068791fb42f58f642bec18b941d5feab10d8"
Jul 21 18:28:45.718827 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.718833    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2dc35135c9d23e6f1dd53e86df6068791fb42f58f642bec18b941d5feab10d8"} err="failed to get container status \"f2dc35135c9d23e6f1dd53e86df6068791fb42f58f642bec18b941d5feab10d8\": rpc error: code = NotFound desc = could not find container \"f2dc35135c9d23e6f1dd53e86df6068791fb42f58f642bec18b941d5feab10d8\": container with ID starting with f2dc35135c9d23e6f1dd53e86df6068791fb42f58f642bec18b941d5feab10d8 not found: ID does not exist"
Jul 21 18:28:45.718968 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.718849    2562 scope.go:117] "RemoveContainer" containerID="fbcef7cd332113ddcd24cc27e949be370a093bc211e5d52d1ac5991f1e08a6fa"
Jul 21 18:28:45.719182 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:28:45.719163    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbcef7cd332113ddcd24cc27e949be370a093bc211e5d52d1ac5991f1e08a6fa\": container with ID starting with fbcef7cd332113ddcd24cc27e949be370a093bc211e5d52d1ac5991f1e08a6fa not found: ID does not exist" containerID="fbcef7cd332113ddcd24cc27e949be370a093bc211e5d52d1ac5991f1e08a6fa"
Jul 21 18:28:45.719360 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.719188    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbcef7cd332113ddcd24cc27e949be370a093bc211e5d52d1ac5991f1e08a6fa"} err="failed to get container status \"fbcef7cd332113ddcd24cc27e949be370a093bc211e5d52d1ac5991f1e08a6fa\": rpc error: code = NotFound desc = could not find container \"fbcef7cd332113ddcd24cc27e949be370a093bc211e5d52d1ac5991f1e08a6fa\": container with ID starting with fbcef7cd332113ddcd24cc27e949be370a093bc211e5d52d1ac5991f1e08a6fa not found: ID does not exist"
Jul 21 18:28:45.719360 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.719203    2562 scope.go:117] "RemoveContainer" containerID="5b2545c086ab63ea412234a5ebfe215e0cfc1dc638724def25cf5bf93cf6ee9d"
Jul 21 18:28:45.719478 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:28:45.719425    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b2545c086ab63ea412234a5ebfe215e0cfc1dc638724def25cf5bf93cf6ee9d\": container with ID starting with 5b2545c086ab63ea412234a5ebfe215e0cfc1dc638724def25cf5bf93cf6ee9d not found: ID does not exist" containerID="5b2545c086ab63ea412234a5ebfe215e0cfc1dc638724def25cf5bf93cf6ee9d"
Jul 21 18:28:45.719478 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.719442    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b2545c086ab63ea412234a5ebfe215e0cfc1dc638724def25cf5bf93cf6ee9d"} err="failed to get container status \"5b2545c086ab63ea412234a5ebfe215e0cfc1dc638724def25cf5bf93cf6ee9d\": rpc error: code = NotFound desc = could not find container \"5b2545c086ab63ea412234a5ebfe215e0cfc1dc638724def25cf5bf93cf6ee9d\": container with ID starting with 5b2545c086ab63ea412234a5ebfe215e0cfc1dc638724def25cf5bf93cf6ee9d not found: ID does not exist"
Jul 21 18:28:45.719478 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:45.719455    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-757d748996-njdv7"]
Jul 21 18:28:46.921230 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:46.921198    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e977334-378c-4cfa-a69a-b50729246cb3" path="/var/lib/kubelet/pods/6e977334-378c-4cfa-a69a-b50729246cb3/volumes"
Jul 21 18:28:48.964046 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964018    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"]
Jul 21 18:28:48.964441 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964319    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6765d34-ec56-4bfe-acd7-ca86865af464" containerName="storage-initializer"
Jul 21 18:28:48.964441 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964330    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6765d34-ec56-4bfe-acd7-ca86865af464" containerName="storage-initializer"
Jul 21 18:28:48.964441 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964338    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e977334-378c-4cfa-a69a-b50729246cb3" containerName="kserve-container"
Jul 21 18:28:48.964441 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964344    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e977334-378c-4cfa-a69a-b50729246cb3" containerName="kserve-container"
Jul 21 18:28:48.964441 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964356    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6765d34-ec56-4bfe-acd7-ca86865af464" containerName="kube-rbac-proxy"
Jul 21 18:28:48.964441 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964361    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6765d34-ec56-4bfe-acd7-ca86865af464" containerName="kube-rbac-proxy"
Jul 21 18:28:48.964441 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964368    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e977334-378c-4cfa-a69a-b50729246cb3" containerName="kube-rbac-proxy"
Jul 21 18:28:48.964441 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964373    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e977334-378c-4cfa-a69a-b50729246cb3" containerName="kube-rbac-proxy"
Jul 21 18:28:48.964441 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964381    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e977334-378c-4cfa-a69a-b50729246cb3" containerName="storage-initializer"
Jul 21 18:28:48.964441 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964387    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e977334-378c-4cfa-a69a-b50729246cb3" containerName="storage-initializer"
Jul 21 18:28:48.964441 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964401    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6765d34-ec56-4bfe-acd7-ca86865af464" containerName="kserve-container"
Jul 21 18:28:48.964441 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964406    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6765d34-ec56-4bfe-acd7-ca86865af464" containerName="kserve-container"
Jul 21 18:28:48.964820 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964453    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6765d34-ec56-4bfe-acd7-ca86865af464" containerName="kube-rbac-proxy"
Jul 21 18:28:48.964820 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964470    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e977334-378c-4cfa-a69a-b50729246cb3" containerName="kube-rbac-proxy"
Jul 21 18:28:48.964820 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964477    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e977334-378c-4cfa-a69a-b50729246cb3" containerName="kserve-container"
Jul 21 18:28:48.964820 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.964483    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6765d34-ec56-4bfe-acd7-ca86865af464" containerName="kserve-container"
Jul 21 18:28:48.968880 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.968863    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:48.971712 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.971673    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:28:48.971712 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.971689    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 21 18:28:48.971907 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.971674    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:28:48.971907 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.971754    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:28:48.971907 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.971808    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:28:48.972801 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.972784    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:28:48.972890 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.972832    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 21 18:28:48.978532 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:48.978511    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"]
Jul 21 18:28:49.010494 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.010471    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/553394ef-c670-4e56-8531-1df85a6d5d34-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:49.010603 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.010503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/553394ef-c670-4e56-8531-1df85a6d5d34-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:49.010676 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.010622    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h98b9\" (UniqueName: \"kubernetes.io/projected/553394ef-c670-4e56-8531-1df85a6d5d34-kube-api-access-h98b9\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:49.010676 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.010668    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/553394ef-c670-4e56-8531-1df85a6d5d34-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:49.010799 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.010688    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/553394ef-c670-4e56-8531-1df85a6d5d34-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:49.111928 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.111899    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h98b9\" (UniqueName: \"kubernetes.io/projected/553394ef-c670-4e56-8531-1df85a6d5d34-kube-api-access-h98b9\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:49.112094 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.111932    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/553394ef-c670-4e56-8531-1df85a6d5d34-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:49.112094 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.111950    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/553394ef-c670-4e56-8531-1df85a6d5d34-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:49.112094 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.111969    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/553394ef-c670-4e56-8531-1df85a6d5d34-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:49.112094 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.111994    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/553394ef-c670-4e56-8531-1df85a6d5d34-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:49.112316 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:28:49.112101    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 21 18:28:49.112316 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:28:49.112169    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/553394ef-c670-4e56-8531-1df85a6d5d34-proxy-tls podName:553394ef-c670-4e56-8531-1df85a6d5d34 nodeName:}" failed. No retries permitted until 2026-07-21 18:28:49.612148278 +0000 UTC m=+3781.183770010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/553394ef-c670-4e56-8531-1df85a6d5d34-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh" (UID: "553394ef-c670-4e56-8531-1df85a6d5d34") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 21 18:28:49.112651 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.112635    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/553394ef-c670-4e56-8531-1df85a6d5d34-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:49.112790 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.112770    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/553394ef-c670-4e56-8531-1df85a6d5d34-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:49.112829 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.112770    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/553394ef-c670-4e56-8531-1df85a6d5d34-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:49.120847 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.120823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h98b9\" (UniqueName: \"kubernetes.io/projected/553394ef-c670-4e56-8531-1df85a6d5d34-kube-api-access-h98b9\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:49.616930 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.616845    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/553394ef-c670-4e56-8531-1df85a6d5d34-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:49.619447 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.619415    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/553394ef-c670-4e56-8531-1df85a6d5d34-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:49.879465 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:49.879381    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:50.016041 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:50.015968    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"]
Jul 21 18:28:50.018490 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:28:50.018465    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553394ef_c670_4e56_8531_1df85a6d5d34.slice/crio-6195200bbd8c0955bd24ba5cb32d99361cc8092a06cd0a723b30fd617e1a63bb WatchSource:0}: Error finding container 6195200bbd8c0955bd24ba5cb32d99361cc8092a06cd0a723b30fd617e1a63bb: Status 404 returned error can't find the container with id 6195200bbd8c0955bd24ba5cb32d99361cc8092a06cd0a723b30fd617e1a63bb
Jul 21 18:28:50.717215 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:50.717180    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh" event={"ID":"553394ef-c670-4e56-8531-1df85a6d5d34","Type":"ContainerStarted","Data":"19e1a9ccf738281dc6d9890e0d9e0371f71cf4fbdbb24b0b9a7ab7ca78e5564a"}
Jul 21 18:28:50.717399 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:50.717240    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh" event={"ID":"553394ef-c670-4e56-8531-1df85a6d5d34","Type":"ContainerStarted","Data":"6195200bbd8c0955bd24ba5cb32d99361cc8092a06cd0a723b30fd617e1a63bb"}
Jul 21 18:28:54.733171 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:54.733134    2562 generic.go:356] "Generic (PLEG): container finished" podID="553394ef-c670-4e56-8531-1df85a6d5d34" containerID="19e1a9ccf738281dc6d9890e0d9e0371f71cf4fbdbb24b0b9a7ab7ca78e5564a" exitCode=0
Jul 21 18:28:54.733557 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:54.733206    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh" event={"ID":"553394ef-c670-4e56-8531-1df85a6d5d34","Type":"ContainerDied","Data":"19e1a9ccf738281dc6d9890e0d9e0371f71cf4fbdbb24b0b9a7ab7ca78e5564a"}
Jul 21 18:28:55.739074 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:55.739041    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh" event={"ID":"553394ef-c670-4e56-8531-1df85a6d5d34","Type":"ContainerStarted","Data":"a677df4c4c724eeccb8badec1b5bfd2f6d18c6fbb1c1a63adaf8bb72408597cf"}
Jul 21 18:28:55.739446 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:55.739118    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh" event={"ID":"553394ef-c670-4e56-8531-1df85a6d5d34","Type":"ContainerStarted","Data":"d2c98a07ac5c6fe08a572d327a33ed6d36ab8b82103499132ab5a97a6813d19a"}
Jul 21 18:28:55.739446 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:55.739348    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:28:55.760923 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:55.760878    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh" podStartSLOduration=4.043351826 podStartE2EDuration="7.760865382s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.717513556s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:28:55.759923241 +0000 UTC m=+3787.331544998" watchObservedRunningTime="2026-07-21 18:28:55.760865382 +0000 UTC m=+3787.332487138"
Jul 21 18:28:56.743104 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:28:56.743053    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:29:02.753707 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:02.753676    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:29:32.792856 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:32.792807    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh" podUID="553394ef-c670-4e56-8531-1df85a6d5d34" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 21 18:29:42.756978 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:42.756949    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:29:49.068396 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:49.068358    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"]
Jul 21 18:29:49.068849 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:49.068604    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh" podUID="553394ef-c670-4e56-8531-1df85a6d5d34" containerName="kserve-container" containerID="cri-o://d2c98a07ac5c6fe08a572d327a33ed6d36ab8b82103499132ab5a97a6813d19a" gracePeriod=30
Jul 21 18:29:49.068849 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:49.068642    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh" podUID="553394ef-c670-4e56-8531-1df85a6d5d34" containerName="kube-rbac-proxy" containerID="cri-o://a677df4c4c724eeccb8badec1b5bfd2f6d18c6fbb1c1a63adaf8bb72408597cf" gracePeriod=30
Jul 21 18:29:49.955881 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:49.955849    2562 generic.go:356] "Generic (PLEG): container finished" podID="553394ef-c670-4e56-8531-1df85a6d5d34" containerID="a677df4c4c724eeccb8badec1b5bfd2f6d18c6fbb1c1a63adaf8bb72408597cf" exitCode=2
Jul 21 18:29:49.956054 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:49.955924    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh" event={"ID":"553394ef-c670-4e56-8531-1df85a6d5d34","Type":"ContainerDied","Data":"a677df4c4c724eeccb8badec1b5bfd2f6d18c6fbb1c1a63adaf8bb72408597cf"}
Jul 21 18:29:52.747727 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:52.747690    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh" podUID="553394ef-c670-4e56-8531-1df85a6d5d34" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Jul 21 18:29:52.754867 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:52.754839    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh" podUID="553394ef-c670-4e56-8531-1df85a6d5d34" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 21 18:29:54.233302 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.233269    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"]
Jul 21 18:29:54.236818 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.236796    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.239367 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.239349    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 21 18:29:54.239474 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.239374    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 21 18:29:54.246474 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.246443    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"]
Jul 21 18:29:54.328982 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.328950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/268f7e13-b82b-478d-ad42-9ca7fbf484de-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.328982 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.328981    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvnwf\" (UniqueName: \"kubernetes.io/projected/268f7e13-b82b-478d-ad42-9ca7fbf484de-kube-api-access-gvnwf\") pod \"isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.329213 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.329048    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/268f7e13-b82b-478d-ad42-9ca7fbf484de-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.329213 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.329153    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/268f7e13-b82b-478d-ad42-9ca7fbf484de-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.329213 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.329200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/268f7e13-b82b-478d-ad42-9ca7fbf484de-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.430694 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.430656    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/268f7e13-b82b-478d-ad42-9ca7fbf484de-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.430871 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.430720    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/268f7e13-b82b-478d-ad42-9ca7fbf484de-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.430871 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.430787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/268f7e13-b82b-478d-ad42-9ca7fbf484de-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.430871 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.430819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvnwf\" (UniqueName: \"kubernetes.io/projected/268f7e13-b82b-478d-ad42-9ca7fbf484de-kube-api-access-gvnwf\") pod \"isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.431041 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.430882    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/268f7e13-b82b-478d-ad42-9ca7fbf484de-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.431392 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.431368    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/268f7e13-b82b-478d-ad42-9ca7fbf484de-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.432011 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.431985    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/268f7e13-b82b-478d-ad42-9ca7fbf484de-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.435135 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.435112    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/268f7e13-b82b-478d-ad42-9ca7fbf484de-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.435714 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.435688    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/268f7e13-b82b-478d-ad42-9ca7fbf484de-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.445004 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.444985    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvnwf\" (UniqueName: \"kubernetes.io/projected/268f7e13-b82b-478d-ad42-9ca7fbf484de-kube-api-access-gvnwf\") pod \"isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.547409 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.547337    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:29:54.680132 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.680110    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"]
Jul 21 18:29:54.681761 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:29:54.681732    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268f7e13_b82b_478d_ad42_9ca7fbf484de.slice/crio-e503fabb1b870fb8a2b69f3eed9b1806ee6f2f318566672162079d8f1b4c0cfe WatchSource:0}: Error finding container e503fabb1b870fb8a2b69f3eed9b1806ee6f2f318566672162079d8f1b4c0cfe: Status 404 returned error can't find the container with id e503fabb1b870fb8a2b69f3eed9b1806ee6f2f318566672162079d8f1b4c0cfe
Jul 21 18:29:54.976943 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.976906    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" event={"ID":"268f7e13-b82b-478d-ad42-9ca7fbf484de","Type":"ContainerStarted","Data":"8609a50e70ec3c242e70fd51260e431c0611bae969607367ea45a91808b3e12b"}
Jul 21 18:29:54.977154 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:54.976972    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" event={"ID":"268f7e13-b82b-478d-ad42-9ca7fbf484de","Type":"ContainerStarted","Data":"e503fabb1b870fb8a2b69f3eed9b1806ee6f2f318566672162079d8f1b4c0cfe"}
Jul 21 18:29:55.921595 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.921567    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:29:55.943234 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.943207    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/553394ef-c670-4e56-8531-1df85a6d5d34-cabundle-cert\") pod \"553394ef-c670-4e56-8531-1df85a6d5d34\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") "
Jul 21 18:29:55.943377 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.943268    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/553394ef-c670-4e56-8531-1df85a6d5d34-kube-rbac-proxy-sar-config\") pod \"553394ef-c670-4e56-8531-1df85a6d5d34\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") "
Jul 21 18:29:55.943377 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.943307    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/553394ef-c670-4e56-8531-1df85a6d5d34-kserve-provision-location\") pod \"553394ef-c670-4e56-8531-1df85a6d5d34\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") "
Jul 21 18:29:55.943377 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.943351    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/553394ef-c670-4e56-8531-1df85a6d5d34-proxy-tls\") pod \"553394ef-c670-4e56-8531-1df85a6d5d34\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") "
Jul 21 18:29:55.943537 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.943383    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h98b9\" (UniqueName: \"kubernetes.io/projected/553394ef-c670-4e56-8531-1df85a6d5d34-kube-api-access-h98b9\") pod \"553394ef-c670-4e56-8531-1df85a6d5d34\" (UID: \"553394ef-c670-4e56-8531-1df85a6d5d34\") "
Jul 21 18:29:55.943613 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.943575    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/553394ef-c670-4e56-8531-1df85a6d5d34-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "553394ef-c670-4e56-8531-1df85a6d5d34" (UID: "553394ef-c670-4e56-8531-1df85a6d5d34"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:29:55.943723 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.943640    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/553394ef-c670-4e56-8531-1df85a6d5d34-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "553394ef-c670-4e56-8531-1df85a6d5d34" (UID: "553394ef-c670-4e56-8531-1df85a6d5d34"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:29:55.943723 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.943682    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/553394ef-c670-4e56-8531-1df85a6d5d34-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:29:55.944010 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.943985    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/553394ef-c670-4e56-8531-1df85a6d5d34-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "553394ef-c670-4e56-8531-1df85a6d5d34" (UID: "553394ef-c670-4e56-8531-1df85a6d5d34"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:29:55.946133 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.946107    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/553394ef-c670-4e56-8531-1df85a6d5d34-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "553394ef-c670-4e56-8531-1df85a6d5d34" (UID: "553394ef-c670-4e56-8531-1df85a6d5d34"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:29:55.946133 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.946122    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/553394ef-c670-4e56-8531-1df85a6d5d34-kube-api-access-h98b9" (OuterVolumeSpecName: "kube-api-access-h98b9") pod "553394ef-c670-4e56-8531-1df85a6d5d34" (UID: "553394ef-c670-4e56-8531-1df85a6d5d34"). InnerVolumeSpecName "kube-api-access-h98b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:29:55.982167 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.982139    2562 generic.go:356] "Generic (PLEG): container finished" podID="553394ef-c670-4e56-8531-1df85a6d5d34" containerID="d2c98a07ac5c6fe08a572d327a33ed6d36ab8b82103499132ab5a97a6813d19a" exitCode=0
Jul 21 18:29:55.982271 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.982198    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"
Jul 21 18:29:55.982271 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.982226    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh" event={"ID":"553394ef-c670-4e56-8531-1df85a6d5d34","Type":"ContainerDied","Data":"d2c98a07ac5c6fe08a572d327a33ed6d36ab8b82103499132ab5a97a6813d19a"}
Jul 21 18:29:55.982389 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.982299    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh" event={"ID":"553394ef-c670-4e56-8531-1df85a6d5d34","Type":"ContainerDied","Data":"6195200bbd8c0955bd24ba5cb32d99361cc8092a06cd0a723b30fd617e1a63bb"}
Jul 21 18:29:55.982389 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.982341    2562 scope.go:117] "RemoveContainer" containerID="a677df4c4c724eeccb8badec1b5bfd2f6d18c6fbb1c1a63adaf8bb72408597cf"
Jul 21 18:29:55.991125 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.991105    2562 scope.go:117] "RemoveContainer" containerID="d2c98a07ac5c6fe08a572d327a33ed6d36ab8b82103499132ab5a97a6813d19a"
Jul 21 18:29:55.998254 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:55.998238    2562 scope.go:117] "RemoveContainer" containerID="19e1a9ccf738281dc6d9890e0d9e0371f71cf4fbdbb24b0b9a7ab7ca78e5564a"
Jul 21 18:29:56.004458 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:56.004430    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"]
Jul 21 18:29:56.006284 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:56.006264    2562 scope.go:117] "RemoveContainer" containerID="a677df4c4c724eeccb8badec1b5bfd2f6d18c6fbb1c1a63adaf8bb72408597cf"
Jul 21 18:29:56.006586 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:29:56.006560    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a677df4c4c724eeccb8badec1b5bfd2f6d18c6fbb1c1a63adaf8bb72408597cf\": container with ID starting with a677df4c4c724eeccb8badec1b5bfd2f6d18c6fbb1c1a63adaf8bb72408597cf not found: ID does not exist" containerID="a677df4c4c724eeccb8badec1b5bfd2f6d18c6fbb1c1a63adaf8bb72408597cf"
Jul 21 18:29:56.006682 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:56.006594    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a677df4c4c724eeccb8badec1b5bfd2f6d18c6fbb1c1a63adaf8bb72408597cf"} err="failed to get container status \"a677df4c4c724eeccb8badec1b5bfd2f6d18c6fbb1c1a63adaf8bb72408597cf\": rpc error: code = NotFound desc = could not find container \"a677df4c4c724eeccb8badec1b5bfd2f6d18c6fbb1c1a63adaf8bb72408597cf\": container with ID starting with a677df4c4c724eeccb8badec1b5bfd2f6d18c6fbb1c1a63adaf8bb72408597cf not found: ID does not exist"
Jul 21 18:29:56.006682 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:56.006618    2562 scope.go:117] "RemoveContainer" containerID="d2c98a07ac5c6fe08a572d327a33ed6d36ab8b82103499132ab5a97a6813d19a"
Jul 21 18:29:56.006936 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:29:56.006916    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c98a07ac5c6fe08a572d327a33ed6d36ab8b82103499132ab5a97a6813d19a\": container with ID starting with d2c98a07ac5c6fe08a572d327a33ed6d36ab8b82103499132ab5a97a6813d19a not found: ID does not exist" containerID="d2c98a07ac5c6fe08a572d327a33ed6d36ab8b82103499132ab5a97a6813d19a"
Jul 21 18:29:56.007018 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:56.006942    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c98a07ac5c6fe08a572d327a33ed6d36ab8b82103499132ab5a97a6813d19a"} err="failed to get container status \"d2c98a07ac5c6fe08a572d327a33ed6d36ab8b82103499132ab5a97a6813d19a\": rpc error: code = NotFound desc = could not find container \"d2c98a07ac5c6fe08a572d327a33ed6d36ab8b82103499132ab5a97a6813d19a\": container with ID starting with d2c98a07ac5c6fe08a572d327a33ed6d36ab8b82103499132ab5a97a6813d19a not found: ID does not exist"
Jul 21 18:29:56.007018 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:56.006959    2562 scope.go:117] "RemoveContainer" containerID="19e1a9ccf738281dc6d9890e0d9e0371f71cf4fbdbb24b0b9a7ab7ca78e5564a"
Jul 21 18:29:56.007490 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:29:56.007466    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e1a9ccf738281dc6d9890e0d9e0371f71cf4fbdbb24b0b9a7ab7ca78e5564a\": container with ID starting with 19e1a9ccf738281dc6d9890e0d9e0371f71cf4fbdbb24b0b9a7ab7ca78e5564a not found: ID does not exist" containerID="19e1a9ccf738281dc6d9890e0d9e0371f71cf4fbdbb24b0b9a7ab7ca78e5564a"
Jul 21 18:29:56.007574 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:56.007499    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e1a9ccf738281dc6d9890e0d9e0371f71cf4fbdbb24b0b9a7ab7ca78e5564a"} err="failed to get container status \"19e1a9ccf738281dc6d9890e0d9e0371f71cf4fbdbb24b0b9a7ab7ca78e5564a\": rpc error: code = NotFound desc = could not find container \"19e1a9ccf738281dc6d9890e0d9e0371f71cf4fbdbb24b0b9a7ab7ca78e5564a\": container with ID starting with 19e1a9ccf738281dc6d9890e0d9e0371f71cf4fbdbb24b0b9a7ab7ca78e5564a not found: ID does not exist"
Jul 21 18:29:56.008233 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:56.008213    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6mvbh"]
Jul 21 18:29:56.044047 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:56.044021    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/553394ef-c670-4e56-8531-1df85a6d5d34-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:29:56.044047 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:56.044043    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/553394ef-c670-4e56-8531-1df85a6d5d34-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:29:56.044225 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:56.044056    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/553394ef-c670-4e56-8531-1df85a6d5d34-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:29:56.044225 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:56.044091    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h98b9\" (UniqueName: \"kubernetes.io/projected/553394ef-c670-4e56-8531-1df85a6d5d34-kube-api-access-h98b9\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:29:56.922153 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:56.922116    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="553394ef-c670-4e56-8531-1df85a6d5d34" path="/var/lib/kubelet/pods/553394ef-c670-4e56-8531-1df85a6d5d34/volumes"
Jul 21 18:29:58.998414 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:58.998383    2562 generic.go:356] "Generic (PLEG): container finished" podID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerID="8609a50e70ec3c242e70fd51260e431c0611bae969607367ea45a91808b3e12b" exitCode=0
Jul 21 18:29:58.998802 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:29:58.998448    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" event={"ID":"268f7e13-b82b-478d-ad42-9ca7fbf484de","Type":"ContainerDied","Data":"8609a50e70ec3c242e70fd51260e431c0611bae969607367ea45a91808b3e12b"}
Jul 21 18:30:00.005341 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:30:00.005308    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" event={"ID":"268f7e13-b82b-478d-ad42-9ca7fbf484de","Type":"ContainerStarted","Data":"336c59ecafdc94b3e834fea9c536402b4ac2f9eca18b7194bd880792b89b8987"}
Jul 21 18:30:00.005722 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:30:00.005368    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" event={"ID":"268f7e13-b82b-478d-ad42-9ca7fbf484de","Type":"ContainerStarted","Data":"eda0e764425632a5d3a634bfef5452e5cd04228ecea31ed5e7739ea67d7d13a7"}
Jul 21 18:30:00.005722 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:30:00.005558    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:30:00.026521 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:30:00.026470    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" podStartSLOduration=2.280686485 podStartE2EDuration="6.026456232s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.745769747s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:30:00.024486457 +0000 UTC m=+3851.596108228" watchObservedRunningTime="2026-07-21 18:30:00.026456232 +0000 UTC m=+3851.598077986"
Jul 21 18:30:01.009950 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:30:01.009917    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:30:01.011167 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:30:01.011142    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 21 18:30:02.015167 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:30:02.015125    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 21 18:30:07.019622 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:30:07.019596    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:30:07.020145 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:30:07.020098    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 21 18:30:17.021019 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:30:17.020937    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 21 18:30:27.020995 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:30:27.020956    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 21 18:30:37.020342 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:30:37.020300    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 21 18:30:47.021106 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:30:47.021047    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 21 18:30:49.156397 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:30:49.156371    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:30:49.161897 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:30:49.161873    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:30:57.021038 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:30:57.020996    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 21 18:31:07.020697 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:07.020662    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 21 18:31:07.917123 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:07.917058    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 21 18:31:17.918988 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:17.918958    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:31:24.448378 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:24.448341    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"]
Jul 21 18:31:24.448851 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:24.448648    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kserve-container" containerID="cri-o://eda0e764425632a5d3a634bfef5452e5cd04228ecea31ed5e7739ea67d7d13a7" gracePeriod=30
Jul 21 18:31:24.448851 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:24.448818    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kube-rbac-proxy" containerID="cri-o://336c59ecafdc94b3e834fea9c536402b4ac2f9eca18b7194bd880792b89b8987" gracePeriod=30
Jul 21 18:31:25.365263 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:25.365227    2562 generic.go:356] "Generic (PLEG): container finished" podID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerID="336c59ecafdc94b3e834fea9c536402b4ac2f9eca18b7194bd880792b89b8987" exitCode=2
Jul 21 18:31:25.365439 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:25.365299    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" event={"ID":"268f7e13-b82b-478d-ad42-9ca7fbf484de","Type":"ContainerDied","Data":"336c59ecafdc94b3e834fea9c536402b4ac2f9eca18b7194bd880792b89b8987"}
Jul 21 18:31:27.016030 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:27.015991    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jul 21 18:31:27.917526 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:27.917485    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 21 18:31:28.787786 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:28.787763    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:31:28.893637 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:28.892633    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/268f7e13-b82b-478d-ad42-9ca7fbf484de-proxy-tls\") pod \"268f7e13-b82b-478d-ad42-9ca7fbf484de\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") "
Jul 21 18:31:28.894577 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:28.893851    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/268f7e13-b82b-478d-ad42-9ca7fbf484de-kube-rbac-proxy-sar-config\") pod \"268f7e13-b82b-478d-ad42-9ca7fbf484de\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") "
Jul 21 18:31:28.894577 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:28.893907    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/268f7e13-b82b-478d-ad42-9ca7fbf484de-kserve-provision-location\") pod \"268f7e13-b82b-478d-ad42-9ca7fbf484de\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") "
Jul 21 18:31:28.894577 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:28.893945    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvnwf\" (UniqueName: \"kubernetes.io/projected/268f7e13-b82b-478d-ad42-9ca7fbf484de-kube-api-access-gvnwf\") pod \"268f7e13-b82b-478d-ad42-9ca7fbf484de\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") "
Jul 21 18:31:28.894577 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:28.893969    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/268f7e13-b82b-478d-ad42-9ca7fbf484de-cabundle-cert\") pod \"268f7e13-b82b-478d-ad42-9ca7fbf484de\" (UID: \"268f7e13-b82b-478d-ad42-9ca7fbf484de\") "
Jul 21 18:31:28.895219 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:28.894962    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268f7e13-b82b-478d-ad42-9ca7fbf484de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "268f7e13-b82b-478d-ad42-9ca7fbf484de" (UID: "268f7e13-b82b-478d-ad42-9ca7fbf484de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:31:28.897944 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:28.895799    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/268f7e13-b82b-478d-ad42-9ca7fbf484de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "268f7e13-b82b-478d-ad42-9ca7fbf484de" (UID: "268f7e13-b82b-478d-ad42-9ca7fbf484de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:31:28.897944 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:28.896095    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268f7e13-b82b-478d-ad42-9ca7fbf484de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "268f7e13-b82b-478d-ad42-9ca7fbf484de" (UID: "268f7e13-b82b-478d-ad42-9ca7fbf484de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:31:28.899771 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:28.898175    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268f7e13-b82b-478d-ad42-9ca7fbf484de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "268f7e13-b82b-478d-ad42-9ca7fbf484de" (UID: "268f7e13-b82b-478d-ad42-9ca7fbf484de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:31:28.899771 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:28.898299    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268f7e13-b82b-478d-ad42-9ca7fbf484de-kube-api-access-gvnwf" (OuterVolumeSpecName: "kube-api-access-gvnwf") pod "268f7e13-b82b-478d-ad42-9ca7fbf484de" (UID: "268f7e13-b82b-478d-ad42-9ca7fbf484de"). InnerVolumeSpecName "kube-api-access-gvnwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:31:28.995359 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:28.995325    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/268f7e13-b82b-478d-ad42-9ca7fbf484de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:31:28.995359 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:28.995356    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/268f7e13-b82b-478d-ad42-9ca7fbf484de-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:31:28.995359 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:28.995366    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvnwf\" (UniqueName: \"kubernetes.io/projected/268f7e13-b82b-478d-ad42-9ca7fbf484de-kube-api-access-gvnwf\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:31:28.995522 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:28.995375    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/268f7e13-b82b-478d-ad42-9ca7fbf484de-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:31:28.995522 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:28.995384    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/268f7e13-b82b-478d-ad42-9ca7fbf484de-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:31:29.383421 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.383386    2562 generic.go:356] "Generic (PLEG): container finished" podID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerID="eda0e764425632a5d3a634bfef5452e5cd04228ecea31ed5e7739ea67d7d13a7" exitCode=0
Jul 21 18:31:29.383594 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.383461    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"
Jul 21 18:31:29.383594 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.383465    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" event={"ID":"268f7e13-b82b-478d-ad42-9ca7fbf484de","Type":"ContainerDied","Data":"eda0e764425632a5d3a634bfef5452e5cd04228ecea31ed5e7739ea67d7d13a7"}
Jul 21 18:31:29.383594 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.383521    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf" event={"ID":"268f7e13-b82b-478d-ad42-9ca7fbf484de","Type":"ContainerDied","Data":"e503fabb1b870fb8a2b69f3eed9b1806ee6f2f318566672162079d8f1b4c0cfe"}
Jul 21 18:31:29.383594 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.383549    2562 scope.go:117] "RemoveContainer" containerID="336c59ecafdc94b3e834fea9c536402b4ac2f9eca18b7194bd880792b89b8987"
Jul 21 18:31:29.392018 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.392004    2562 scope.go:117] "RemoveContainer" containerID="eda0e764425632a5d3a634bfef5452e5cd04228ecea31ed5e7739ea67d7d13a7"
Jul 21 18:31:29.399263 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.399244    2562 scope.go:117] "RemoveContainer" containerID="8609a50e70ec3c242e70fd51260e431c0611bae969607367ea45a91808b3e12b"
Jul 21 18:31:29.401935 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.401911    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"]
Jul 21 18:31:29.406554 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.406535    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c9f7f6496-cmwcf"]
Jul 21 18:31:29.407128 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.407112    2562 scope.go:117] "RemoveContainer" containerID="336c59ecafdc94b3e834fea9c536402b4ac2f9eca18b7194bd880792b89b8987"
Jul 21 18:31:29.407359 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:31:29.407344    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"336c59ecafdc94b3e834fea9c536402b4ac2f9eca18b7194bd880792b89b8987\": container with ID starting with 336c59ecafdc94b3e834fea9c536402b4ac2f9eca18b7194bd880792b89b8987 not found: ID does not exist" containerID="336c59ecafdc94b3e834fea9c536402b4ac2f9eca18b7194bd880792b89b8987"
Jul 21 18:31:29.407417 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.407366    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336c59ecafdc94b3e834fea9c536402b4ac2f9eca18b7194bd880792b89b8987"} err="failed to get container status \"336c59ecafdc94b3e834fea9c536402b4ac2f9eca18b7194bd880792b89b8987\": rpc error: code = NotFound desc = could not find container \"336c59ecafdc94b3e834fea9c536402b4ac2f9eca18b7194bd880792b89b8987\": container with ID starting with 336c59ecafdc94b3e834fea9c536402b4ac2f9eca18b7194bd880792b89b8987 not found: ID does not exist"
Jul 21 18:31:29.407417 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.407380    2562 scope.go:117] "RemoveContainer" containerID="eda0e764425632a5d3a634bfef5452e5cd04228ecea31ed5e7739ea67d7d13a7"
Jul 21 18:31:29.407592 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:31:29.407578    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda0e764425632a5d3a634bfef5452e5cd04228ecea31ed5e7739ea67d7d13a7\": container with ID starting with eda0e764425632a5d3a634bfef5452e5cd04228ecea31ed5e7739ea67d7d13a7 not found: ID does not exist" containerID="eda0e764425632a5d3a634bfef5452e5cd04228ecea31ed5e7739ea67d7d13a7"
Jul 21 18:31:29.407636 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.407596    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda0e764425632a5d3a634bfef5452e5cd04228ecea31ed5e7739ea67d7d13a7"} err="failed to get container status \"eda0e764425632a5d3a634bfef5452e5cd04228ecea31ed5e7739ea67d7d13a7\": rpc error: code = NotFound desc = could not find container \"eda0e764425632a5d3a634bfef5452e5cd04228ecea31ed5e7739ea67d7d13a7\": container with ID starting with eda0e764425632a5d3a634bfef5452e5cd04228ecea31ed5e7739ea67d7d13a7 not found: ID does not exist"
Jul 21 18:31:29.407636 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.407608    2562 scope.go:117] "RemoveContainer" containerID="8609a50e70ec3c242e70fd51260e431c0611bae969607367ea45a91808b3e12b"
Jul 21 18:31:29.407810 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:31:29.407788    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8609a50e70ec3c242e70fd51260e431c0611bae969607367ea45a91808b3e12b\": container with ID starting with 8609a50e70ec3c242e70fd51260e431c0611bae969607367ea45a91808b3e12b not found: ID does not exist" containerID="8609a50e70ec3c242e70fd51260e431c0611bae969607367ea45a91808b3e12b"
Jul 21 18:31:29.407863 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.407819    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8609a50e70ec3c242e70fd51260e431c0611bae969607367ea45a91808b3e12b"} err="failed to get container status \"8609a50e70ec3c242e70fd51260e431c0611bae969607367ea45a91808b3e12b\": rpc error: code = NotFound desc = could not find container \"8609a50e70ec3c242e70fd51260e431c0611bae969607367ea45a91808b3e12b\": container with ID starting with 8609a50e70ec3c242e70fd51260e431c0611bae969607367ea45a91808b3e12b not found: ID does not exist"
Jul 21 18:31:29.613764 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.613734    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"]
Jul 21 18:31:29.614073 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.614051    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="553394ef-c670-4e56-8531-1df85a6d5d34" containerName="kserve-container"
Jul 21 18:31:29.614194 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.614079    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="553394ef-c670-4e56-8531-1df85a6d5d34" containerName="kserve-container"
Jul 21 18:31:29.614194 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.614093    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kserve-container"
Jul 21 18:31:29.614194 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.614099    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kserve-container"
Jul 21 18:31:29.614194 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.614108    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="553394ef-c670-4e56-8531-1df85a6d5d34" containerName="storage-initializer"
Jul 21 18:31:29.614194 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.614114    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="553394ef-c670-4e56-8531-1df85a6d5d34" containerName="storage-initializer"
Jul 21 18:31:29.614194 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.614123    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="storage-initializer"
Jul 21 18:31:29.614194 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.614128    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="storage-initializer"
Jul 21 18:31:29.614194 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.614141    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="553394ef-c670-4e56-8531-1df85a6d5d34" containerName="kube-rbac-proxy"
Jul 21 18:31:29.614194 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.614146    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="553394ef-c670-4e56-8531-1df85a6d5d34" containerName="kube-rbac-proxy"
Jul 21 18:31:29.614194 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.614152    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kube-rbac-proxy"
Jul 21 18:31:29.614194 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.614157    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kube-rbac-proxy"
Jul 21 18:31:29.614194 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.614200    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="553394ef-c670-4e56-8531-1df85a6d5d34" containerName="kserve-container"
Jul 21 18:31:29.614566 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.614208    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kserve-container"
Jul 21 18:31:29.614566 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.614214    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" containerName="kube-rbac-proxy"
Jul 21 18:31:29.614566 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.614221    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="553394ef-c670-4e56-8531-1df85a6d5d34" containerName="kube-rbac-proxy"
Jul 21 18:31:29.618592 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.618575    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:29.621252 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.621232    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:31:29.621340 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.621279    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 21 18:31:29.621340 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.621300    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:31:29.621557 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.621536    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:31:29.621602 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.621593    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:31:29.621640 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.621560    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 21 18:31:29.621692 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.621628    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:31:29.628748 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.628726    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"]
Jul 21 18:31:29.701766 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.701741    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f187b13b-c53b-43ab-8d59-75f92113d413-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:29.701941 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.701774    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f187b13b-c53b-43ab-8d59-75f92113d413-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:29.701941 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.701799    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f187b13b-c53b-43ab-8d59-75f92113d413-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:29.701941 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.701817    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rd4w\" (UniqueName: \"kubernetes.io/projected/f187b13b-c53b-43ab-8d59-75f92113d413-kube-api-access-9rd4w\") pod \"isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:29.701941 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.701908    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f187b13b-c53b-43ab-8d59-75f92113d413-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:29.803281 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.803247    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f187b13b-c53b-43ab-8d59-75f92113d413-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:29.803720 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.803310    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f187b13b-c53b-43ab-8d59-75f92113d413-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:29.803720 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.803339    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f187b13b-c53b-43ab-8d59-75f92113d413-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:29.803720 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.803372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f187b13b-c53b-43ab-8d59-75f92113d413-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:29.803720 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.803399    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rd4w\" (UniqueName: \"kubernetes.io/projected/f187b13b-c53b-43ab-8d59-75f92113d413-kube-api-access-9rd4w\") pod \"isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:29.803914 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.803725    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f187b13b-c53b-43ab-8d59-75f92113d413-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:29.803976 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.803958    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f187b13b-c53b-43ab-8d59-75f92113d413-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:29.804016 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.803973    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f187b13b-c53b-43ab-8d59-75f92113d413-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:29.805816 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.805796    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f187b13b-c53b-43ab-8d59-75f92113d413-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:29.811840 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.811819    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rd4w\" (UniqueName: \"kubernetes.io/projected/f187b13b-c53b-43ab-8d59-75f92113d413-kube-api-access-9rd4w\") pod \"isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:29.929376 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:29.929348    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:30.058897 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:30.058877    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"]
Jul 21 18:31:30.061324 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:31:30.061294    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf187b13b_c53b_43ab_8d59_75f92113d413.slice/crio-5eff7227ec4d95474cb54e40a083d314fc40be02261f71e260248d20553cf383 WatchSource:0}: Error finding container 5eff7227ec4d95474cb54e40a083d314fc40be02261f71e260248d20553cf383: Status 404 returned error can't find the container with id 5eff7227ec4d95474cb54e40a083d314fc40be02261f71e260248d20553cf383
Jul 21 18:31:30.389365 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:30.389279    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" event={"ID":"f187b13b-c53b-43ab-8d59-75f92113d413","Type":"ContainerStarted","Data":"f5ca6d81ca6ba6b510393a8fcf71e23867782258bd7781ed31d3798b47e5f09b"}
Jul 21 18:31:30.389365 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:30.389342    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" event={"ID":"f187b13b-c53b-43ab-8d59-75f92113d413","Type":"ContainerStarted","Data":"5eff7227ec4d95474cb54e40a083d314fc40be02261f71e260248d20553cf383"}
Jul 21 18:31:30.921443 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:30.921410    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="268f7e13-b82b-478d-ad42-9ca7fbf484de" path="/var/lib/kubelet/pods/268f7e13-b82b-478d-ad42-9ca7fbf484de/volumes"
Jul 21 18:31:34.409291 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:34.409259    2562 generic.go:356] "Generic (PLEG): container finished" podID="f187b13b-c53b-43ab-8d59-75f92113d413" containerID="f5ca6d81ca6ba6b510393a8fcf71e23867782258bd7781ed31d3798b47e5f09b" exitCode=0
Jul 21 18:31:34.409664 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:34.409334    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" event={"ID":"f187b13b-c53b-43ab-8d59-75f92113d413","Type":"ContainerDied","Data":"f5ca6d81ca6ba6b510393a8fcf71e23867782258bd7781ed31d3798b47e5f09b"}
Jul 21 18:31:35.415580 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:35.415545    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" event={"ID":"f187b13b-c53b-43ab-8d59-75f92113d413","Type":"ContainerStarted","Data":"06bd08c5ad1a7140fc6f12228a2fe20a0b7df44801a30123fffca0ccdd7379eb"}
Jul 21 18:31:35.415963 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:35.415608    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" event={"ID":"f187b13b-c53b-43ab-8d59-75f92113d413","Type":"ContainerStarted","Data":"909c975beca57d8cf27547e1f050e55de7c23641e99d95e3eddccb613059faba"}
Jul 21 18:31:35.415963 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:35.415884    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:35.416098 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:35.416041    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:35.417390 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:35.417364    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 21 18:31:35.436171 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:35.436129    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" podStartSLOduration=2.848277627 podStartE2EDuration="6.436116003s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.587838376s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:31:35.434352414 +0000 UTC m=+3947.005974169" watchObservedRunningTime="2026-07-21 18:31:35.436116003 +0000 UTC m=+3947.007737758"
Jul 21 18:31:36.420168 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:36.420128    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 21 18:31:37.424928 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:37.424891    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 21 18:31:42.429731 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:42.429700    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:31:42.430329 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:42.430302    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 21 18:31:52.430918 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:31:52.430877    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 21 18:32:02.430789 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:32:02.430749    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 21 18:32:12.430844 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:32:12.430807    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 21 18:32:22.431271 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:32:22.431231    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 21 18:32:32.430772 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:32:32.430732    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 21 18:32:42.430720 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:32:42.430680    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 21 18:32:42.916805 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:32:42.916764    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 21 18:32:52.922051 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:32:52.922019    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:32:59.782532 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:32:59.782492    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"]
Jul 21 18:32:59.782987 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:32:59.782811    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kserve-container" containerID="cri-o://909c975beca57d8cf27547e1f050e55de7c23641e99d95e3eddccb613059faba" gracePeriod=30
Jul 21 18:32:59.782987 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:32:59.782867    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kube-rbac-proxy" containerID="cri-o://06bd08c5ad1a7140fc6f12228a2fe20a0b7df44801a30123fffca0ccdd7379eb" gracePeriod=30
Jul 21 18:33:00.772283 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:00.772252    2562 generic.go:356] "Generic (PLEG): container finished" podID="f187b13b-c53b-43ab-8d59-75f92113d413" containerID="06bd08c5ad1a7140fc6f12228a2fe20a0b7df44801a30123fffca0ccdd7379eb" exitCode=2
Jul 21 18:33:00.772283 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:00.772289    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" event={"ID":"f187b13b-c53b-43ab-8d59-75f92113d413","Type":"ContainerDied","Data":"06bd08c5ad1a7140fc6f12228a2fe20a0b7df44801a30123fffca0ccdd7379eb"}
Jul 21 18:33:02.425815 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:02.425774    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Jul 21 18:33:02.916656 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:02.916621    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 21 18:33:04.227431 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.227409    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:33:04.257413 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.257347    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f187b13b-c53b-43ab-8d59-75f92113d413-kube-rbac-proxy-sar-config\") pod \"f187b13b-c53b-43ab-8d59-75f92113d413\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") "
Jul 21 18:33:04.257413 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.257392    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f187b13b-c53b-43ab-8d59-75f92113d413-cabundle-cert\") pod \"f187b13b-c53b-43ab-8d59-75f92113d413\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") "
Jul 21 18:33:04.257602 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.257418    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f187b13b-c53b-43ab-8d59-75f92113d413-kserve-provision-location\") pod \"f187b13b-c53b-43ab-8d59-75f92113d413\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") "
Jul 21 18:33:04.257602 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.257460    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rd4w\" (UniqueName: \"kubernetes.io/projected/f187b13b-c53b-43ab-8d59-75f92113d413-kube-api-access-9rd4w\") pod \"f187b13b-c53b-43ab-8d59-75f92113d413\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") "
Jul 21 18:33:04.257602 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.257493    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f187b13b-c53b-43ab-8d59-75f92113d413-proxy-tls\") pod \"f187b13b-c53b-43ab-8d59-75f92113d413\" (UID: \"f187b13b-c53b-43ab-8d59-75f92113d413\") "
Jul 21 18:33:04.257775 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.257660    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f187b13b-c53b-43ab-8d59-75f92113d413-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f187b13b-c53b-43ab-8d59-75f92113d413" (UID: "f187b13b-c53b-43ab-8d59-75f92113d413"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:33:04.257775 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.257735    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f187b13b-c53b-43ab-8d59-75f92113d413-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f187b13b-c53b-43ab-8d59-75f92113d413" (UID: "f187b13b-c53b-43ab-8d59-75f92113d413"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:33:04.257775 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.257770    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f187b13b-c53b-43ab-8d59-75f92113d413-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f187b13b-c53b-43ab-8d59-75f92113d413" (UID: "f187b13b-c53b-43ab-8d59-75f92113d413"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:33:04.259616 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.259588    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f187b13b-c53b-43ab-8d59-75f92113d413-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f187b13b-c53b-43ab-8d59-75f92113d413" (UID: "f187b13b-c53b-43ab-8d59-75f92113d413"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:33:04.259788 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.259767    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f187b13b-c53b-43ab-8d59-75f92113d413-kube-api-access-9rd4w" (OuterVolumeSpecName: "kube-api-access-9rd4w") pod "f187b13b-c53b-43ab-8d59-75f92113d413" (UID: "f187b13b-c53b-43ab-8d59-75f92113d413"). InnerVolumeSpecName "kube-api-access-9rd4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:33:04.358607 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.358567    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rd4w\" (UniqueName: \"kubernetes.io/projected/f187b13b-c53b-43ab-8d59-75f92113d413-kube-api-access-9rd4w\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:33:04.358607 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.358602    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f187b13b-c53b-43ab-8d59-75f92113d413-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:33:04.358607 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.358612    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f187b13b-c53b-43ab-8d59-75f92113d413-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:33:04.358607 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.358621    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f187b13b-c53b-43ab-8d59-75f92113d413-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:33:04.358840 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.358629    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f187b13b-c53b-43ab-8d59-75f92113d413-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:33:04.790723 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.790680    2562 generic.go:356] "Generic (PLEG): container finished" podID="f187b13b-c53b-43ab-8d59-75f92113d413" containerID="909c975beca57d8cf27547e1f050e55de7c23641e99d95e3eddccb613059faba" exitCode=0
Jul 21 18:33:04.790723 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.790721    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"
Jul 21 18:33:04.790915 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.790755    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" event={"ID":"f187b13b-c53b-43ab-8d59-75f92113d413","Type":"ContainerDied","Data":"909c975beca57d8cf27547e1f050e55de7c23641e99d95e3eddccb613059faba"}
Jul 21 18:33:04.790915 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.790805    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj" event={"ID":"f187b13b-c53b-43ab-8d59-75f92113d413","Type":"ContainerDied","Data":"5eff7227ec4d95474cb54e40a083d314fc40be02261f71e260248d20553cf383"}
Jul 21 18:33:04.790915 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.790831    2562 scope.go:117] "RemoveContainer" containerID="06bd08c5ad1a7140fc6f12228a2fe20a0b7df44801a30123fffca0ccdd7379eb"
Jul 21 18:33:04.801402 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.801080    2562 scope.go:117] "RemoveContainer" containerID="909c975beca57d8cf27547e1f050e55de7c23641e99d95e3eddccb613059faba"
Jul 21 18:33:04.810035 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.810017    2562 scope.go:117] "RemoveContainer" containerID="f5ca6d81ca6ba6b510393a8fcf71e23867782258bd7781ed31d3798b47e5f09b"
Jul 21 18:33:04.816717 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.816694    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"]
Jul 21 18:33:04.818259 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.818238    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-87d9ffc8f-z7wzj"]
Jul 21 18:33:04.818466 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.818437    2562 scope.go:117] "RemoveContainer" containerID="06bd08c5ad1a7140fc6f12228a2fe20a0b7df44801a30123fffca0ccdd7379eb"
Jul 21 18:33:04.818763 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:33:04.818743    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06bd08c5ad1a7140fc6f12228a2fe20a0b7df44801a30123fffca0ccdd7379eb\": container with ID starting with 06bd08c5ad1a7140fc6f12228a2fe20a0b7df44801a30123fffca0ccdd7379eb not found: ID does not exist" containerID="06bd08c5ad1a7140fc6f12228a2fe20a0b7df44801a30123fffca0ccdd7379eb"
Jul 21 18:33:04.818817 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.818771    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bd08c5ad1a7140fc6f12228a2fe20a0b7df44801a30123fffca0ccdd7379eb"} err="failed to get container status \"06bd08c5ad1a7140fc6f12228a2fe20a0b7df44801a30123fffca0ccdd7379eb\": rpc error: code = NotFound desc = could not find container \"06bd08c5ad1a7140fc6f12228a2fe20a0b7df44801a30123fffca0ccdd7379eb\": container with ID starting with 06bd08c5ad1a7140fc6f12228a2fe20a0b7df44801a30123fffca0ccdd7379eb not found: ID does not exist"
Jul 21 18:33:04.818817 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.818788    2562 scope.go:117] "RemoveContainer" containerID="909c975beca57d8cf27547e1f050e55de7c23641e99d95e3eddccb613059faba"
Jul 21 18:33:04.819023 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:33:04.819002    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909c975beca57d8cf27547e1f050e55de7c23641e99d95e3eddccb613059faba\": container with ID starting with 909c975beca57d8cf27547e1f050e55de7c23641e99d95e3eddccb613059faba not found: ID does not exist" containerID="909c975beca57d8cf27547e1f050e55de7c23641e99d95e3eddccb613059faba"
Jul 21 18:33:04.819126 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.819034    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909c975beca57d8cf27547e1f050e55de7c23641e99d95e3eddccb613059faba"} err="failed to get container status \"909c975beca57d8cf27547e1f050e55de7c23641e99d95e3eddccb613059faba\": rpc error: code = NotFound desc = could not find container \"909c975beca57d8cf27547e1f050e55de7c23641e99d95e3eddccb613059faba\": container with ID starting with 909c975beca57d8cf27547e1f050e55de7c23641e99d95e3eddccb613059faba not found: ID does not exist"
Jul 21 18:33:04.819126 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.819056    2562 scope.go:117] "RemoveContainer" containerID="f5ca6d81ca6ba6b510393a8fcf71e23867782258bd7781ed31d3798b47e5f09b"
Jul 21 18:33:04.819342 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:33:04.819325    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ca6d81ca6ba6b510393a8fcf71e23867782258bd7781ed31d3798b47e5f09b\": container with ID starting with f5ca6d81ca6ba6b510393a8fcf71e23867782258bd7781ed31d3798b47e5f09b not found: ID does not exist" containerID="f5ca6d81ca6ba6b510393a8fcf71e23867782258bd7781ed31d3798b47e5f09b"
Jul 21 18:33:04.819384 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.819346    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ca6d81ca6ba6b510393a8fcf71e23867782258bd7781ed31d3798b47e5f09b"} err="failed to get container status \"f5ca6d81ca6ba6b510393a8fcf71e23867782258bd7781ed31d3798b47e5f09b\": rpc error: code = NotFound desc = could not find container \"f5ca6d81ca6ba6b510393a8fcf71e23867782258bd7781ed31d3798b47e5f09b\": container with ID starting with f5ca6d81ca6ba6b510393a8fcf71e23867782258bd7781ed31d3798b47e5f09b not found: ID does not exist"
Jul 21 18:33:04.922372 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:33:04.922346    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" path="/var/lib/kubelet/pods/f187b13b-c53b-43ab-8d59-75f92113d413/volumes"
Jul 21 18:34:36.851910 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.851877    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"]
Jul 21 18:34:36.852353 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.852221    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kserve-container"
Jul 21 18:34:36.852353 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.852232    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kserve-container"
Jul 21 18:34:36.852353 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.852243    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="storage-initializer"
Jul 21 18:34:36.852353 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.852248    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="storage-initializer"
Jul 21 18:34:36.852353 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.852255    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kube-rbac-proxy"
Jul 21 18:34:36.852353 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.852260    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kube-rbac-proxy"
Jul 21 18:34:36.852353 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.852310    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kserve-container"
Jul 21 18:34:36.852353 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.852322    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f187b13b-c53b-43ab-8d59-75f92113d413" containerName="kube-rbac-proxy"
Jul 21 18:34:36.855614 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.855587    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:36.858405 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.858370    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:34:36.858405 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.858365    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Jul 21 18:34:36.858405 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.858404    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Jul 21 18:34:36.858642 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.858441    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:34:36.858642 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.858400    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:34:36.858735 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.858696    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:34:36.859355 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.859343    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:34:36.866988 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.866964    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"]
Jul 21 18:34:36.976863 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.976832    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30d348ab-ce84-4980-bd28-e360bac00caf-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-gkn9m\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:36.977000 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.976867    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30d348ab-ce84-4980-bd28-e360bac00caf-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-gkn9m\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:36.977000 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.976893    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30d348ab-ce84-4980-bd28-e360bac00caf-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-gkn9m\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:36.977000 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.976917    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30d348ab-ce84-4980-bd28-e360bac00caf-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-gkn9m\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:36.977000 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:36.976975    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vmsb\" (UniqueName: \"kubernetes.io/projected/30d348ab-ce84-4980-bd28-e360bac00caf-kube-api-access-6vmsb\") pod \"isvc-triton-predictor-5d7db58676-gkn9m\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:37.077470 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:37.077446    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30d348ab-ce84-4980-bd28-e360bac00caf-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-gkn9m\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:37.077629 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:37.077489    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30d348ab-ce84-4980-bd28-e360bac00caf-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-gkn9m\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:37.077629 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:37.077540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30d348ab-ce84-4980-bd28-e360bac00caf-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-gkn9m\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:37.077629 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:37.077587    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vmsb\" (UniqueName: \"kubernetes.io/projected/30d348ab-ce84-4980-bd28-e360bac00caf-kube-api-access-6vmsb\") pod \"isvc-triton-predictor-5d7db58676-gkn9m\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:37.077818 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:37.077651    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30d348ab-ce84-4980-bd28-e360bac00caf-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-gkn9m\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:37.077818 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:34:37.077787    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Jul 21 18:34:37.077928 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:34:37.077843    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d348ab-ce84-4980-bd28-e360bac00caf-proxy-tls podName:30d348ab-ce84-4980-bd28-e360bac00caf nodeName:}" failed. No retries permitted until 2026-07-21 18:34:37.577823111 +0000 UTC m=+4129.149444846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/30d348ab-ce84-4980-bd28-e360bac00caf-proxy-tls") pod "isvc-triton-predictor-5d7db58676-gkn9m" (UID: "30d348ab-ce84-4980-bd28-e360bac00caf") : secret "isvc-triton-predictor-serving-cert" not found
Jul 21 18:34:37.077995 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:37.077976    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30d348ab-ce84-4980-bd28-e360bac00caf-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-gkn9m\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:37.078225 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:37.078205    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30d348ab-ce84-4980-bd28-e360bac00caf-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-gkn9m\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:37.078425 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:37.078404    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30d348ab-ce84-4980-bd28-e360bac00caf-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-gkn9m\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:37.086471 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:37.086452    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vmsb\" (UniqueName: \"kubernetes.io/projected/30d348ab-ce84-4980-bd28-e360bac00caf-kube-api-access-6vmsb\") pod \"isvc-triton-predictor-5d7db58676-gkn9m\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:37.581656 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:37.581624    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30d348ab-ce84-4980-bd28-e360bac00caf-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-gkn9m\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:37.584120 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:37.584102    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30d348ab-ce84-4980-bd28-e360bac00caf-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-gkn9m\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:37.766829 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:37.766794    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:34:37.903836 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:37.903779    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"]
Jul 21 18:34:37.905963 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:34:37.905940    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d348ab_ce84_4980_bd28_e360bac00caf.slice/crio-4f9f39e0ab62c7a5bcfed707cf1db52b46983d44dde7a11d422d4a534e85e18a WatchSource:0}: Error finding container 4f9f39e0ab62c7a5bcfed707cf1db52b46983d44dde7a11d422d4a534e85e18a: Status 404 returned error can't find the container with id 4f9f39e0ab62c7a5bcfed707cf1db52b46983d44dde7a11d422d4a534e85e18a
Jul 21 18:34:38.175471 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:38.175433    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m" event={"ID":"30d348ab-ce84-4980-bd28-e360bac00caf","Type":"ContainerStarted","Data":"f7ea4db93ae052d6c9b5511bf0f818826e60152ea4bd75b091833de8a66cbfe7"}
Jul 21 18:34:38.175636 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:38.175504    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m" event={"ID":"30d348ab-ce84-4980-bd28-e360bac00caf","Type":"ContainerStarted","Data":"4f9f39e0ab62c7a5bcfed707cf1db52b46983d44dde7a11d422d4a534e85e18a"}
Jul 21 18:34:42.192554 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:42.192521    2562 generic.go:356] "Generic (PLEG): container finished" podID="30d348ab-ce84-4980-bd28-e360bac00caf" containerID="f7ea4db93ae052d6c9b5511bf0f818826e60152ea4bd75b091833de8a66cbfe7" exitCode=0
Jul 21 18:34:42.192958 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:42.192559    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m" event={"ID":"30d348ab-ce84-4980-bd28-e360bac00caf","Type":"ContainerDied","Data":"f7ea4db93ae052d6c9b5511bf0f818826e60152ea4bd75b091833de8a66cbfe7"}
Jul 21 18:34:42.193722 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:34:42.193693    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 18:36:35.733101 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:35.733008    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:36:35.733634 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:35.733206    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:36:37.778591 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:37.778552    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m" event={"ID":"30d348ab-ce84-4980-bd28-e360bac00caf","Type":"ContainerStarted","Data":"fde04218e613f96eea7aac949fa60b9849b4fba2e970c9957a317f046e778c64"}
Jul 21 18:36:37.779105 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:37.778623    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m" event={"ID":"30d348ab-ce84-4980-bd28-e360bac00caf","Type":"ContainerStarted","Data":"2f00438fc539913bbedbf80d93c996452b8d7810c07aa712d3634ae1357619ef"}
Jul 21 18:36:37.779105 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:37.778722    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:36:37.808952 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:37.808893    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m" podStartSLOduration=3.505301797 podStartE2EDuration="2m1.808879091s" totalImagesPullingTime="1m54.504612748s" totalInitContainerRuntime="3.798964546s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:36:37.808164318 +0000 UTC m=+4249.379786097" watchObservedRunningTime="2026-07-21 18:36:37.808879091 +0000 UTC m=+4249.380500846"
Jul 21 18:36:38.783251 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:38.783216    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:36:38.784682 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:38.784653    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m" podUID="30d348ab-ce84-4980-bd28-e360bac00caf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 21 18:36:39.787242 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:39.787200    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m" podUID="30d348ab-ce84-4980-bd28-e360bac00caf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 21 18:36:44.792368 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:44.792336    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:36:44.793361 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:44.793340    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:36:49.146001 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:49.145966    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"]
Jul 21 18:36:49.146541 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:49.146248    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m" podUID="30d348ab-ce84-4980-bd28-e360bac00caf" containerName="kserve-container" containerID="cri-o://2f00438fc539913bbedbf80d93c996452b8d7810c07aa712d3634ae1357619ef" gracePeriod=30
Jul 21 18:36:49.146541 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:49.146288    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m" podUID="30d348ab-ce84-4980-bd28-e360bac00caf" containerName="kube-rbac-proxy" containerID="cri-o://fde04218e613f96eea7aac949fa60b9849b4fba2e970c9957a317f046e778c64" gracePeriod=30
Jul 21 18:36:49.787255 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:49.787222    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m" podUID="30d348ab-ce84-4980-bd28-e360bac00caf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Jul 21 18:36:49.831275 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:49.831247    2562 generic.go:356] "Generic (PLEG): container finished" podID="30d348ab-ce84-4980-bd28-e360bac00caf" containerID="fde04218e613f96eea7aac949fa60b9849b4fba2e970c9957a317f046e778c64" exitCode=2
Jul 21 18:36:49.831415 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:49.831325    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m" event={"ID":"30d348ab-ce84-4980-bd28-e360bac00caf","Type":"ContainerDied","Data":"fde04218e613f96eea7aac949fa60b9849b4fba2e970c9957a317f046e778c64"}
Jul 21 18:36:51.506198 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.506178    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:36:51.588337 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.588253    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30d348ab-ce84-4980-bd28-e360bac00caf-kube-rbac-proxy-sar-config\") pod \"30d348ab-ce84-4980-bd28-e360bac00caf\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") "
Jul 21 18:36:51.588337 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.588326    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vmsb\" (UniqueName: \"kubernetes.io/projected/30d348ab-ce84-4980-bd28-e360bac00caf-kube-api-access-6vmsb\") pod \"30d348ab-ce84-4980-bd28-e360bac00caf\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") "
Jul 21 18:36:51.588552 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.588368    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30d348ab-ce84-4980-bd28-e360bac00caf-proxy-tls\") pod \"30d348ab-ce84-4980-bd28-e360bac00caf\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") "
Jul 21 18:36:51.588552 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.588428    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30d348ab-ce84-4980-bd28-e360bac00caf-cabundle-cert\") pod \"30d348ab-ce84-4980-bd28-e360bac00caf\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") "
Jul 21 18:36:51.588552 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.588458    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30d348ab-ce84-4980-bd28-e360bac00caf-kserve-provision-location\") pod \"30d348ab-ce84-4980-bd28-e360bac00caf\" (UID: \"30d348ab-ce84-4980-bd28-e360bac00caf\") "
Jul 21 18:36:51.588700 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.588589    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d348ab-ce84-4980-bd28-e360bac00caf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "30d348ab-ce84-4980-bd28-e360bac00caf" (UID: "30d348ab-ce84-4980-bd28-e360bac00caf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:36:51.588700 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.588695    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30d348ab-ce84-4980-bd28-e360bac00caf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:36:51.588781 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.588768    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d348ab-ce84-4980-bd28-e360bac00caf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "30d348ab-ce84-4980-bd28-e360bac00caf" (UID: "30d348ab-ce84-4980-bd28-e360bac00caf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:36:51.588962 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.588940    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d348ab-ce84-4980-bd28-e360bac00caf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30d348ab-ce84-4980-bd28-e360bac00caf" (UID: "30d348ab-ce84-4980-bd28-e360bac00caf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:36:51.590606 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.590584    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d348ab-ce84-4980-bd28-e360bac00caf-kube-api-access-6vmsb" (OuterVolumeSpecName: "kube-api-access-6vmsb") pod "30d348ab-ce84-4980-bd28-e360bac00caf" (UID: "30d348ab-ce84-4980-bd28-e360bac00caf"). InnerVolumeSpecName "kube-api-access-6vmsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:36:51.590661 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.590627    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d348ab-ce84-4980-bd28-e360bac00caf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30d348ab-ce84-4980-bd28-e360bac00caf" (UID: "30d348ab-ce84-4980-bd28-e360bac00caf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:36:51.689827 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.689796    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vmsb\" (UniqueName: \"kubernetes.io/projected/30d348ab-ce84-4980-bd28-e360bac00caf-kube-api-access-6vmsb\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:36:51.689827 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.689821    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30d348ab-ce84-4980-bd28-e360bac00caf-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:36:51.689995 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.689834    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30d348ab-ce84-4980-bd28-e360bac00caf-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:36:51.689995 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.689846    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30d348ab-ce84-4980-bd28-e360bac00caf-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:36:51.841534 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.841450    2562 generic.go:356] "Generic (PLEG): container finished" podID="30d348ab-ce84-4980-bd28-e360bac00caf" containerID="2f00438fc539913bbedbf80d93c996452b8d7810c07aa712d3634ae1357619ef" exitCode=0
Jul 21 18:36:51.841534 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.841506    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"
Jul 21 18:36:51.841534 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.841530    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m" event={"ID":"30d348ab-ce84-4980-bd28-e360bac00caf","Type":"ContainerDied","Data":"2f00438fc539913bbedbf80d93c996452b8d7810c07aa712d3634ae1357619ef"}
Jul 21 18:36:51.841747 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.841577    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m" event={"ID":"30d348ab-ce84-4980-bd28-e360bac00caf","Type":"ContainerDied","Data":"4f9f39e0ab62c7a5bcfed707cf1db52b46983d44dde7a11d422d4a534e85e18a"}
Jul 21 18:36:51.841747 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.841606    2562 scope.go:117] "RemoveContainer" containerID="fde04218e613f96eea7aac949fa60b9849b4fba2e970c9957a317f046e778c64"
Jul 21 18:36:51.859611 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.859586    2562 scope.go:117] "RemoveContainer" containerID="2f00438fc539913bbedbf80d93c996452b8d7810c07aa712d3634ae1357619ef"
Jul 21 18:36:51.868202 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.868180    2562 scope.go:117] "RemoveContainer" containerID="f7ea4db93ae052d6c9b5511bf0f818826e60152ea4bd75b091833de8a66cbfe7"
Jul 21 18:36:51.871415 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.871394    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"]
Jul 21 18:36:51.874695 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.874666    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gkn9m"]
Jul 21 18:36:51.877389 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.877372    2562 scope.go:117] "RemoveContainer" containerID="fde04218e613f96eea7aac949fa60b9849b4fba2e970c9957a317f046e778c64"
Jul 21 18:36:51.877682 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:36:51.877661    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde04218e613f96eea7aac949fa60b9849b4fba2e970c9957a317f046e778c64\": container with ID starting with fde04218e613f96eea7aac949fa60b9849b4fba2e970c9957a317f046e778c64 not found: ID does not exist" containerID="fde04218e613f96eea7aac949fa60b9849b4fba2e970c9957a317f046e778c64"
Jul 21 18:36:51.877814 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.877693    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde04218e613f96eea7aac949fa60b9849b4fba2e970c9957a317f046e778c64"} err="failed to get container status \"fde04218e613f96eea7aac949fa60b9849b4fba2e970c9957a317f046e778c64\": rpc error: code = NotFound desc = could not find container \"fde04218e613f96eea7aac949fa60b9849b4fba2e970c9957a317f046e778c64\": container with ID starting with fde04218e613f96eea7aac949fa60b9849b4fba2e970c9957a317f046e778c64 not found: ID does not exist"
Jul 21 18:36:51.877814 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.877716    2562 scope.go:117] "RemoveContainer" containerID="2f00438fc539913bbedbf80d93c996452b8d7810c07aa712d3634ae1357619ef"
Jul 21 18:36:51.877969 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:36:51.877951    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f00438fc539913bbedbf80d93c996452b8d7810c07aa712d3634ae1357619ef\": container with ID starting with 2f00438fc539913bbedbf80d93c996452b8d7810c07aa712d3634ae1357619ef not found: ID does not exist" containerID="2f00438fc539913bbedbf80d93c996452b8d7810c07aa712d3634ae1357619ef"
Jul 21 18:36:51.878016 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.877976    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f00438fc539913bbedbf80d93c996452b8d7810c07aa712d3634ae1357619ef"} err="failed to get container status \"2f00438fc539913bbedbf80d93c996452b8d7810c07aa712d3634ae1357619ef\": rpc error: code = NotFound desc = could not find container \"2f00438fc539913bbedbf80d93c996452b8d7810c07aa712d3634ae1357619ef\": container with ID starting with 2f00438fc539913bbedbf80d93c996452b8d7810c07aa712d3634ae1357619ef not found: ID does not exist"
Jul 21 18:36:51.878016 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.877992    2562 scope.go:117] "RemoveContainer" containerID="f7ea4db93ae052d6c9b5511bf0f818826e60152ea4bd75b091833de8a66cbfe7"
Jul 21 18:36:51.878277 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:36:51.878259    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ea4db93ae052d6c9b5511bf0f818826e60152ea4bd75b091833de8a66cbfe7\": container with ID starting with f7ea4db93ae052d6c9b5511bf0f818826e60152ea4bd75b091833de8a66cbfe7 not found: ID does not exist" containerID="f7ea4db93ae052d6c9b5511bf0f818826e60152ea4bd75b091833de8a66cbfe7"
Jul 21 18:36:51.878338 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:51.878286    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ea4db93ae052d6c9b5511bf0f818826e60152ea4bd75b091833de8a66cbfe7"} err="failed to get container status \"f7ea4db93ae052d6c9b5511bf0f818826e60152ea4bd75b091833de8a66cbfe7\": rpc error: code = NotFound desc = could not find container \"f7ea4db93ae052d6c9b5511bf0f818826e60152ea4bd75b091833de8a66cbfe7\": container with ID starting with f7ea4db93ae052d6c9b5511bf0f818826e60152ea4bd75b091833de8a66cbfe7 not found: ID does not exist"
Jul 21 18:36:52.921113 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:36:52.921057    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30d348ab-ce84-4980-bd28-e360bac00caf" path="/var/lib/kubelet/pods/30d348ab-ce84-4980-bd28-e360bac00caf/volumes"
Jul 21 18:38:29.579676 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.579638    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"]
Jul 21 18:38:29.580126 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.579995    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30d348ab-ce84-4980-bd28-e360bac00caf" containerName="kserve-container"
Jul 21 18:38:29.580126 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.580008    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="30d348ab-ce84-4980-bd28-e360bac00caf" containerName="kserve-container"
Jul 21 18:38:29.580126 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.580023    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30d348ab-ce84-4980-bd28-e360bac00caf" containerName="storage-initializer"
Jul 21 18:38:29.580126 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.580029    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="30d348ab-ce84-4980-bd28-e360bac00caf" containerName="storage-initializer"
Jul 21 18:38:29.580126 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.580038    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30d348ab-ce84-4980-bd28-e360bac00caf" containerName="kube-rbac-proxy"
Jul 21 18:38:29.580126 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.580047    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="30d348ab-ce84-4980-bd28-e360bac00caf" containerName="kube-rbac-proxy"
Jul 21 18:38:29.580126 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.580122    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="30d348ab-ce84-4980-bd28-e360bac00caf" containerName="kube-rbac-proxy"
Jul 21 18:38:29.580357 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.580133    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="30d348ab-ce84-4980-bd28-e360bac00caf" containerName="kserve-container"
Jul 21 18:38:29.582325 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.582306    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:29.588080 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.588046    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jul 21 18:38:29.588210 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.588150    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:38:29.588210 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.588177    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 21 18:38:29.588210 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.588188    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:38:29.588210 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.588210    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:38:29.588440 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.588300    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:38:29.588440 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.588415    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:38:29.602831 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.602809    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"]
Jul 21 18:38:29.644439 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.644414    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c5b3c61-b2af-4c46-bb1e-780008c20561-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:29.644567 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.644453    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c5b3c61-b2af-4c46-bb1e-780008c20561-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:29.644567 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.644475    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c5b3c61-b2af-4c46-bb1e-780008c20561-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:29.644567 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.644506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rkqg\" (UniqueName: \"kubernetes.io/projected/2c5b3c61-b2af-4c46-bb1e-780008c20561-kube-api-access-6rkqg\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:29.644891 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.644638    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c5b3c61-b2af-4c46-bb1e-780008c20561-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:29.745724 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.745701    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c5b3c61-b2af-4c46-bb1e-780008c20561-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:29.745851 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.745736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c5b3c61-b2af-4c46-bb1e-780008c20561-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:29.745851 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.745766    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rkqg\" (UniqueName: \"kubernetes.io/projected/2c5b3c61-b2af-4c46-bb1e-780008c20561-kube-api-access-6rkqg\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:29.745851 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.745825    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c5b3c61-b2af-4c46-bb1e-780008c20561-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:29.746021 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.745865    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c5b3c61-b2af-4c46-bb1e-780008c20561-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:29.746463 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.746437    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c5b3c61-b2af-4c46-bb1e-780008c20561-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:29.746738 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.746506    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c5b3c61-b2af-4c46-bb1e-780008c20561-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:29.746738 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.746651    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c5b3c61-b2af-4c46-bb1e-780008c20561-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:29.748343 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.748314    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c5b3c61-b2af-4c46-bb1e-780008c20561-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:29.768719 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.768693    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rkqg\" (UniqueName: \"kubernetes.io/projected/2c5b3c61-b2af-4c46-bb1e-780008c20561-kube-api-access-6rkqg\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:29.893535 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:29.893459    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:30.025963 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:30.025936    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"]
Jul 21 18:38:30.027970 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:38:30.027941    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5b3c61_b2af_4c46_bb1e_780008c20561.slice/crio-82c3c13f7be302859f1336de973a98133c672ee43ac219574a425a805d8ff621 WatchSource:0}: Error finding container 82c3c13f7be302859f1336de973a98133c672ee43ac219574a425a805d8ff621: Status 404 returned error can't find the container with id 82c3c13f7be302859f1336de973a98133c672ee43ac219574a425a805d8ff621
Jul 21 18:38:30.234164 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:30.234127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh" event={"ID":"2c5b3c61-b2af-4c46-bb1e-780008c20561","Type":"ContainerStarted","Data":"97b41f9c3cb171c978177c290f3b68a0e7db12b4f24dfdaa623a54c91f3eaff0"}
Jul 21 18:38:30.234337 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:30.234183    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh" event={"ID":"2c5b3c61-b2af-4c46-bb1e-780008c20561","Type":"ContainerStarted","Data":"82c3c13f7be302859f1336de973a98133c672ee43ac219574a425a805d8ff621"}
Jul 21 18:38:34.251262 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:34.251232    2562 generic.go:356] "Generic (PLEG): container finished" podID="2c5b3c61-b2af-4c46-bb1e-780008c20561" containerID="97b41f9c3cb171c978177c290f3b68a0e7db12b4f24dfdaa623a54c91f3eaff0" exitCode=0
Jul 21 18:38:34.251623 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:34.251311    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh" event={"ID":"2c5b3c61-b2af-4c46-bb1e-780008c20561","Type":"ContainerDied","Data":"97b41f9c3cb171c978177c290f3b68a0e7db12b4f24dfdaa623a54c91f3eaff0"}
Jul 21 18:38:35.256700 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:35.256663    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh" event={"ID":"2c5b3c61-b2af-4c46-bb1e-780008c20561","Type":"ContainerStarted","Data":"897c61ab6f716911f6674ea23c643127e94306f952e4708db55dd21c49d98c98"}
Jul 21 18:38:35.257211 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:35.256730    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh" event={"ID":"2c5b3c61-b2af-4c46-bb1e-780008c20561","Type":"ContainerStarted","Data":"bdab61e21465179511ff7ce63ba46e0bf9048a52e148d87b525c82c69bcc58a3"}
Jul 21 18:38:35.257211 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:35.256935    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:35.257211 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:35.256972    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:38:35.277180 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:35.277136    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh" podStartSLOduration=2.489395256 podStartE2EDuration="6.277122437s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.787727181s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:38:35.275634992 +0000 UTC m=+4366.847256761" watchObservedRunningTime="2026-07-21 18:38:35.277122437 +0000 UTC m=+4366.848744191"
Jul 21 18:38:41.266126 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:38:41.266096    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:39:11.270209 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:11.270125    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:39:19.617500 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:19.617468    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"]
Jul 21 18:39:19.617927 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:19.617753    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh" podUID="2c5b3c61-b2af-4c46-bb1e-780008c20561" containerName="kserve-container" containerID="cri-o://bdab61e21465179511ff7ce63ba46e0bf9048a52e148d87b525c82c69bcc58a3" gracePeriod=30
Jul 21 18:39:19.617927 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:19.617805    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh" podUID="2c5b3c61-b2af-4c46-bb1e-780008c20561" containerName="kube-rbac-proxy" containerID="cri-o://897c61ab6f716911f6674ea23c643127e94306f952e4708db55dd21c49d98c98" gracePeriod=30
Jul 21 18:39:20.435763 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:20.435727    2562 generic.go:356] "Generic (PLEG): container finished" podID="2c5b3c61-b2af-4c46-bb1e-780008c20561" containerID="897c61ab6f716911f6674ea23c643127e94306f952e4708db55dd21c49d98c98" exitCode=2
Jul 21 18:39:20.435934 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:20.435797    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh" event={"ID":"2c5b3c61-b2af-4c46-bb1e-780008c20561","Type":"ContainerDied","Data":"897c61ab6f716911f6674ea23c643127e94306f952e4708db55dd21c49d98c98"}
Jul 21 18:39:21.262040 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:21.261996    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh" podUID="2c5b3c61-b2af-4c46-bb1e-780008c20561" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Jul 21 18:39:24.794929 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.794895    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"]
Jul 21 18:39:24.797673 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.797651    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:24.800221 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.800197    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 21 18:39:24.800336 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.800200    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jul 21 18:39:24.809465 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.809441    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"]
Jul 21 18:39:24.887444 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.887410    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2scj\" (UniqueName: \"kubernetes.io/projected/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kube-api-access-n2scj\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-csv8g\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:24.887444 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.887447    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-csv8g\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:24.887650 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.887465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc9d26a9-d89b-40a5-a1c8-4878691fc740-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-csv8g\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:24.887650 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.887502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc9d26a9-d89b-40a5-a1c8-4878691fc740-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-csv8g\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:24.887650 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.887556    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-csv8g\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:24.988378 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.988340    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-csv8g\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:24.988578 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.988404    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2scj\" (UniqueName: \"kubernetes.io/projected/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kube-api-access-n2scj\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-csv8g\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:24.988578 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.988461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-csv8g\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:24.988578 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.988508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc9d26a9-d89b-40a5-a1c8-4878691fc740-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-csv8g\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:24.988578 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.988546    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc9d26a9-d89b-40a5-a1c8-4878691fc740-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-csv8g\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:24.988981 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.988959    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-csv8g\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:24.989182 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.989163    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-csv8g\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:24.989247 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.989206    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc9d26a9-d89b-40a5-a1c8-4878691fc740-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-csv8g\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:24.991112 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.991089    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc9d26a9-d89b-40a5-a1c8-4878691fc740-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-csv8g\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:24.996695 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:24.996676    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2scj\" (UniqueName: \"kubernetes.io/projected/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kube-api-access-n2scj\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-csv8g\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:25.107910 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.107819    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:25.245175 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.245053    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"]
Jul 21 18:39:25.247792 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:39:25.247768    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9d26a9_d89b_40a5_a1c8_4878691fc740.slice/crio-339672f365ba62c34c9838a0a0f50c7a6aa2a408187aea632e4380b34960a9a7 WatchSource:0}: Error finding container 339672f365ba62c34c9838a0a0f50c7a6aa2a408187aea632e4380b34960a9a7: Status 404 returned error can't find the container with id 339672f365ba62c34c9838a0a0f50c7a6aa2a408187aea632e4380b34960a9a7
Jul 21 18:39:25.457573 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.457529    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g" event={"ID":"fc9d26a9-d89b-40a5-a1c8-4878691fc740","Type":"ContainerStarted","Data":"1de03354dc312e857c09b247872ec83ad9ad0c2210108b7407bb2b66b82154a0"}
Jul 21 18:39:25.457761 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.457614    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g" event={"ID":"fc9d26a9-d89b-40a5-a1c8-4878691fc740","Type":"ContainerStarted","Data":"339672f365ba62c34c9838a0a0f50c7a6aa2a408187aea632e4380b34960a9a7"}
Jul 21 18:39:25.652550 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.652525    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:39:25.695702 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.695679    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c5b3c61-b2af-4c46-bb1e-780008c20561-cabundle-cert\") pod \"2c5b3c61-b2af-4c46-bb1e-780008c20561\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") "
Jul 21 18:39:25.695857 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.695711    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rkqg\" (UniqueName: \"kubernetes.io/projected/2c5b3c61-b2af-4c46-bb1e-780008c20561-kube-api-access-6rkqg\") pod \"2c5b3c61-b2af-4c46-bb1e-780008c20561\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") "
Jul 21 18:39:25.695857 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.695735    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c5b3c61-b2af-4c46-bb1e-780008c20561-kube-rbac-proxy-sar-config\") pod \"2c5b3c61-b2af-4c46-bb1e-780008c20561\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") "
Jul 21 18:39:25.695857 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.695761    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c5b3c61-b2af-4c46-bb1e-780008c20561-proxy-tls\") pod \"2c5b3c61-b2af-4c46-bb1e-780008c20561\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") "
Jul 21 18:39:25.695857 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.695847    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c5b3c61-b2af-4c46-bb1e-780008c20561-kserve-provision-location\") pod \"2c5b3c61-b2af-4c46-bb1e-780008c20561\" (UID: \"2c5b3c61-b2af-4c46-bb1e-780008c20561\") "
Jul 21 18:39:25.696168 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.696143    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5b3c61-b2af-4c46-bb1e-780008c20561-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c5b3c61-b2af-4c46-bb1e-780008c20561" (UID: "2c5b3c61-b2af-4c46-bb1e-780008c20561"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:39:25.696168 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.696152    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5b3c61-b2af-4c46-bb1e-780008c20561-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c5b3c61-b2af-4c46-bb1e-780008c20561" (UID: "2c5b3c61-b2af-4c46-bb1e-780008c20561"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:39:25.696327 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.696305    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5b3c61-b2af-4c46-bb1e-780008c20561-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c5b3c61-b2af-4c46-bb1e-780008c20561" (UID: "2c5b3c61-b2af-4c46-bb1e-780008c20561"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:39:25.697995 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.697975    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5b3c61-b2af-4c46-bb1e-780008c20561-kube-api-access-6rkqg" (OuterVolumeSpecName: "kube-api-access-6rkqg") pod "2c5b3c61-b2af-4c46-bb1e-780008c20561" (UID: "2c5b3c61-b2af-4c46-bb1e-780008c20561"). InnerVolumeSpecName "kube-api-access-6rkqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:39:25.697995 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.697993    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c5b3c61-b2af-4c46-bb1e-780008c20561-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c5b3c61-b2af-4c46-bb1e-780008c20561" (UID: "2c5b3c61-b2af-4c46-bb1e-780008c20561"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:39:25.796925 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.796856    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c5b3c61-b2af-4c46-bb1e-780008c20561-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:39:25.796925 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.796880    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c5b3c61-b2af-4c46-bb1e-780008c20561-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:39:25.796925 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.796890    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rkqg\" (UniqueName: \"kubernetes.io/projected/2c5b3c61-b2af-4c46-bb1e-780008c20561-kube-api-access-6rkqg\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:39:25.796925 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.796899    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c5b3c61-b2af-4c46-bb1e-780008c20561-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:39:25.796925 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:25.796908    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c5b3c61-b2af-4c46-bb1e-780008c20561-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:39:26.463037 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:26.463007    2562 generic.go:356] "Generic (PLEG): container finished" podID="2c5b3c61-b2af-4c46-bb1e-780008c20561" containerID="bdab61e21465179511ff7ce63ba46e0bf9048a52e148d87b525c82c69bcc58a3" exitCode=0
Jul 21 18:39:26.463224 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:26.463093    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"
Jul 21 18:39:26.463224 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:26.463089    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh" event={"ID":"2c5b3c61-b2af-4c46-bb1e-780008c20561","Type":"ContainerDied","Data":"bdab61e21465179511ff7ce63ba46e0bf9048a52e148d87b525c82c69bcc58a3"}
Jul 21 18:39:26.463316 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:26.463242    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh" event={"ID":"2c5b3c61-b2af-4c46-bb1e-780008c20561","Type":"ContainerDied","Data":"82c3c13f7be302859f1336de973a98133c672ee43ac219574a425a805d8ff621"}
Jul 21 18:39:26.463316 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:26.463268    2562 scope.go:117] "RemoveContainer" containerID="897c61ab6f716911f6674ea23c643127e94306f952e4708db55dd21c49d98c98"
Jul 21 18:39:26.472201 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:26.472190    2562 scope.go:117] "RemoveContainer" containerID="bdab61e21465179511ff7ce63ba46e0bf9048a52e148d87b525c82c69bcc58a3"
Jul 21 18:39:26.480013 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:26.479996    2562 scope.go:117] "RemoveContainer" containerID="97b41f9c3cb171c978177c290f3b68a0e7db12b4f24dfdaa623a54c91f3eaff0"
Jul 21 18:39:26.487311 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:26.487286    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"]
Jul 21 18:39:26.488343 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:26.488329    2562 scope.go:117] "RemoveContainer" containerID="897c61ab6f716911f6674ea23c643127e94306f952e4708db55dd21c49d98c98"
Jul 21 18:39:26.488633 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:39:26.488614    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897c61ab6f716911f6674ea23c643127e94306f952e4708db55dd21c49d98c98\": container with ID starting with 897c61ab6f716911f6674ea23c643127e94306f952e4708db55dd21c49d98c98 not found: ID does not exist" containerID="897c61ab6f716911f6674ea23c643127e94306f952e4708db55dd21c49d98c98"
Jul 21 18:39:26.488709 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:26.488643    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897c61ab6f716911f6674ea23c643127e94306f952e4708db55dd21c49d98c98"} err="failed to get container status \"897c61ab6f716911f6674ea23c643127e94306f952e4708db55dd21c49d98c98\": rpc error: code = NotFound desc = could not find container \"897c61ab6f716911f6674ea23c643127e94306f952e4708db55dd21c49d98c98\": container with ID starting with 897c61ab6f716911f6674ea23c643127e94306f952e4708db55dd21c49d98c98 not found: ID does not exist"
Jul 21 18:39:26.488709 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:26.488660    2562 scope.go:117] "RemoveContainer" containerID="bdab61e21465179511ff7ce63ba46e0bf9048a52e148d87b525c82c69bcc58a3"
Jul 21 18:39:26.488957 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:39:26.488937    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdab61e21465179511ff7ce63ba46e0bf9048a52e148d87b525c82c69bcc58a3\": container with ID starting with bdab61e21465179511ff7ce63ba46e0bf9048a52e148d87b525c82c69bcc58a3 not found: ID does not exist" containerID="bdab61e21465179511ff7ce63ba46e0bf9048a52e148d87b525c82c69bcc58a3"
Jul 21 18:39:26.489009 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:26.488965    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdab61e21465179511ff7ce63ba46e0bf9048a52e148d87b525c82c69bcc58a3"} err="failed to get container status \"bdab61e21465179511ff7ce63ba46e0bf9048a52e148d87b525c82c69bcc58a3\": rpc error: code = NotFound desc = could not find container \"bdab61e21465179511ff7ce63ba46e0bf9048a52e148d87b525c82c69bcc58a3\": container with ID starting with bdab61e21465179511ff7ce63ba46e0bf9048a52e148d87b525c82c69bcc58a3 not found: ID does not exist"
Jul 21 18:39:26.489009 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:26.488982    2562 scope.go:117] "RemoveContainer" containerID="97b41f9c3cb171c978177c290f3b68a0e7db12b4f24dfdaa623a54c91f3eaff0"
Jul 21 18:39:26.489250 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:39:26.489234    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b41f9c3cb171c978177c290f3b68a0e7db12b4f24dfdaa623a54c91f3eaff0\": container with ID starting with 97b41f9c3cb171c978177c290f3b68a0e7db12b4f24dfdaa623a54c91f3eaff0 not found: ID does not exist" containerID="97b41f9c3cb171c978177c290f3b68a0e7db12b4f24dfdaa623a54c91f3eaff0"
Jul 21 18:39:26.489306 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:26.489255    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b41f9c3cb171c978177c290f3b68a0e7db12b4f24dfdaa623a54c91f3eaff0"} err="failed to get container status \"97b41f9c3cb171c978177c290f3b68a0e7db12b4f24dfdaa623a54c91f3eaff0\": rpc error: code = NotFound desc = could not find container \"97b41f9c3cb171c978177c290f3b68a0e7db12b4f24dfdaa623a54c91f3eaff0\": container with ID starting with 97b41f9c3cb171c978177c290f3b68a0e7db12b4f24dfdaa623a54c91f3eaff0 not found: ID does not exist"
Jul 21 18:39:26.493342 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:26.493321    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jbqlh"]
Jul 21 18:39:26.920878 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:26.920847    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c5b3c61-b2af-4c46-bb1e-780008c20561" path="/var/lib/kubelet/pods/2c5b3c61-b2af-4c46-bb1e-780008c20561/volumes"
Jul 21 18:39:29.477815 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:29.477785    2562 generic.go:356] "Generic (PLEG): container finished" podID="fc9d26a9-d89b-40a5-a1c8-4878691fc740" containerID="1de03354dc312e857c09b247872ec83ad9ad0c2210108b7407bb2b66b82154a0" exitCode=0
Jul 21 18:39:29.478188 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:29.477860    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g" event={"ID":"fc9d26a9-d89b-40a5-a1c8-4878691fc740","Type":"ContainerDied","Data":"1de03354dc312e857c09b247872ec83ad9ad0c2210108b7407bb2b66b82154a0"}
Jul 21 18:39:30.484401 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:30.484367    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g" event={"ID":"fc9d26a9-d89b-40a5-a1c8-4878691fc740","Type":"ContainerStarted","Data":"08a0f57507ae7935fdaba42f3f4ca197cd0bf7efd50e1fb70ab931ce6c5ae0af"}
Jul 21 18:39:30.484815 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:30.484429    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g" event={"ID":"fc9d26a9-d89b-40a5-a1c8-4878691fc740","Type":"ContainerStarted","Data":"9454d9dd7858760160c74a47ac1e06a767b86c330fc731d0b31af4d86b81557e"}
Jul 21 18:39:30.484815 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:30.484632    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:30.505240 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:30.505193    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g" podStartSLOduration=2.550937169 podStartE2EDuration="6.505179777s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.954242608s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:39:30.504365457 +0000 UTC m=+4422.075987260" watchObservedRunningTime="2026-07-21 18:39:30.505179777 +0000 UTC m=+4422.076801532"
Jul 21 18:39:31.489610 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:31.489583    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:39:37.500082 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:39:37.500035    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:40:07.505040 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:07.505009    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:40:14.902516 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:14.902483    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"]
Jul 21 18:40:14.902927 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:14.902709    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g" podUID="fc9d26a9-d89b-40a5-a1c8-4878691fc740" containerName="kserve-container" containerID="cri-o://9454d9dd7858760160c74a47ac1e06a767b86c330fc731d0b31af4d86b81557e" gracePeriod=30
Jul 21 18:40:14.902927 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:14.902758    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g" podUID="fc9d26a9-d89b-40a5-a1c8-4878691fc740" containerName="kube-rbac-proxy" containerID="cri-o://08a0f57507ae7935fdaba42f3f4ca197cd0bf7efd50e1fb70ab931ce6c5ae0af" gracePeriod=30
Jul 21 18:40:15.671881 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:15.671848    2562 generic.go:356] "Generic (PLEG): container finished" podID="fc9d26a9-d89b-40a5-a1c8-4878691fc740" containerID="08a0f57507ae7935fdaba42f3f4ca197cd0bf7efd50e1fb70ab931ce6c5ae0af" exitCode=2
Jul 21 18:40:15.672096 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:15.671906    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g" event={"ID":"fc9d26a9-d89b-40a5-a1c8-4878691fc740","Type":"ContainerDied","Data":"08a0f57507ae7935fdaba42f3f4ca197cd0bf7efd50e1fb70ab931ce6c5ae0af"}
Jul 21 18:40:17.495132 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:17.495089    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g" podUID="fc9d26a9-d89b-40a5-a1c8-4878691fc740" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Jul 21 18:40:17.500583 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:17.500552    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g" podUID="fc9d26a9-d89b-40a5-a1c8-4878691fc740" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.53:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 21 18:40:20.854272 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:20.854251    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:40:20.928709 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:20.928682    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc9d26a9-d89b-40a5-a1c8-4878691fc740-proxy-tls\") pod \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") "
Jul 21 18:40:20.928854 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:20.928735    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kserve-provision-location\") pod \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") "
Jul 21 18:40:20.928854 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:20.928758    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2scj\" (UniqueName: \"kubernetes.io/projected/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kube-api-access-n2scj\") pod \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") "
Jul 21 18:40:20.928854 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:20.928789    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc9d26a9-d89b-40a5-a1c8-4878691fc740-cabundle-cert\") pod \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") "
Jul 21 18:40:20.928854 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:20.928804    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kube-rbac-proxy-sar-config\") pod \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\" (UID: \"fc9d26a9-d89b-40a5-a1c8-4878691fc740\") "
Jul 21 18:40:20.929135 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:20.929107    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc9d26a9-d89b-40a5-a1c8-4878691fc740" (UID: "fc9d26a9-d89b-40a5-a1c8-4878691fc740"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:40:20.929253 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:20.929229    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc9d26a9-d89b-40a5-a1c8-4878691fc740-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc9d26a9-d89b-40a5-a1c8-4878691fc740" (UID: "fc9d26a9-d89b-40a5-a1c8-4878691fc740"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:40:20.929314 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:20.929273    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc9d26a9-d89b-40a5-a1c8-4878691fc740" (UID: "fc9d26a9-d89b-40a5-a1c8-4878691fc740"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:40:20.931013 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:20.930988    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc9d26a9-d89b-40a5-a1c8-4878691fc740-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc9d26a9-d89b-40a5-a1c8-4878691fc740" (UID: "fc9d26a9-d89b-40a5-a1c8-4878691fc740"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:40:20.931140 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:20.930991    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kube-api-access-n2scj" (OuterVolumeSpecName: "kube-api-access-n2scj") pod "fc9d26a9-d89b-40a5-a1c8-4878691fc740" (UID: "fc9d26a9-d89b-40a5-a1c8-4878691fc740"). InnerVolumeSpecName "kube-api-access-n2scj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:40:21.029761 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.029678    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc9d26a9-d89b-40a5-a1c8-4878691fc740-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:40:21.029761 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.029706    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:40:21.029761 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.029721    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2scj\" (UniqueName: \"kubernetes.io/projected/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kube-api-access-n2scj\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:40:21.029761 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.029735    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc9d26a9-d89b-40a5-a1c8-4878691fc740-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:40:21.029761 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.029746    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc9d26a9-d89b-40a5-a1c8-4878691fc740-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:40:21.696500 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.696458    2562 generic.go:356] "Generic (PLEG): container finished" podID="fc9d26a9-d89b-40a5-a1c8-4878691fc740" containerID="9454d9dd7858760160c74a47ac1e06a767b86c330fc731d0b31af4d86b81557e" exitCode=0
Jul 21 18:40:21.696675 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.696523    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"
Jul 21 18:40:21.696675 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.696533    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g" event={"ID":"fc9d26a9-d89b-40a5-a1c8-4878691fc740","Type":"ContainerDied","Data":"9454d9dd7858760160c74a47ac1e06a767b86c330fc731d0b31af4d86b81557e"}
Jul 21 18:40:21.696675 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.696580    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g" event={"ID":"fc9d26a9-d89b-40a5-a1c8-4878691fc740","Type":"ContainerDied","Data":"339672f365ba62c34c9838a0a0f50c7a6aa2a408187aea632e4380b34960a9a7"}
Jul 21 18:40:21.696675 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.696606    2562 scope.go:117] "RemoveContainer" containerID="08a0f57507ae7935fdaba42f3f4ca197cd0bf7efd50e1fb70ab931ce6c5ae0af"
Jul 21 18:40:21.706056 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.706039    2562 scope.go:117] "RemoveContainer" containerID="9454d9dd7858760160c74a47ac1e06a767b86c330fc731d0b31af4d86b81557e"
Jul 21 18:40:21.713652 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.713636    2562 scope.go:117] "RemoveContainer" containerID="1de03354dc312e857c09b247872ec83ad9ad0c2210108b7407bb2b66b82154a0"
Jul 21 18:40:21.718421 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.718398    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"]
Jul 21 18:40:21.721898 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.721877    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-csv8g"]
Jul 21 18:40:21.722300 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.722163    2562 scope.go:117] "RemoveContainer" containerID="08a0f57507ae7935fdaba42f3f4ca197cd0bf7efd50e1fb70ab931ce6c5ae0af"
Jul 21 18:40:21.722464 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:40:21.722445    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a0f57507ae7935fdaba42f3f4ca197cd0bf7efd50e1fb70ab931ce6c5ae0af\": container with ID starting with 08a0f57507ae7935fdaba42f3f4ca197cd0bf7efd50e1fb70ab931ce6c5ae0af not found: ID does not exist" containerID="08a0f57507ae7935fdaba42f3f4ca197cd0bf7efd50e1fb70ab931ce6c5ae0af"
Jul 21 18:40:21.722502 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.722472    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a0f57507ae7935fdaba42f3f4ca197cd0bf7efd50e1fb70ab931ce6c5ae0af"} err="failed to get container status \"08a0f57507ae7935fdaba42f3f4ca197cd0bf7efd50e1fb70ab931ce6c5ae0af\": rpc error: code = NotFound desc = could not find container \"08a0f57507ae7935fdaba42f3f4ca197cd0bf7efd50e1fb70ab931ce6c5ae0af\": container with ID starting with 08a0f57507ae7935fdaba42f3f4ca197cd0bf7efd50e1fb70ab931ce6c5ae0af not found: ID does not exist"
Jul 21 18:40:21.722502 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.722491    2562 scope.go:117] "RemoveContainer" containerID="9454d9dd7858760160c74a47ac1e06a767b86c330fc731d0b31af4d86b81557e"
Jul 21 18:40:21.722731 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:40:21.722714    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9454d9dd7858760160c74a47ac1e06a767b86c330fc731d0b31af4d86b81557e\": container with ID starting with 9454d9dd7858760160c74a47ac1e06a767b86c330fc731d0b31af4d86b81557e not found: ID does not exist" containerID="9454d9dd7858760160c74a47ac1e06a767b86c330fc731d0b31af4d86b81557e"
Jul 21 18:40:21.722777 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.722737    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9454d9dd7858760160c74a47ac1e06a767b86c330fc731d0b31af4d86b81557e"} err="failed to get container status \"9454d9dd7858760160c74a47ac1e06a767b86c330fc731d0b31af4d86b81557e\": rpc error: code = NotFound desc = could not find container \"9454d9dd7858760160c74a47ac1e06a767b86c330fc731d0b31af4d86b81557e\": container with ID starting with 9454d9dd7858760160c74a47ac1e06a767b86c330fc731d0b31af4d86b81557e not found: ID does not exist"
Jul 21 18:40:21.722777 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.722753    2562 scope.go:117] "RemoveContainer" containerID="1de03354dc312e857c09b247872ec83ad9ad0c2210108b7407bb2b66b82154a0"
Jul 21 18:40:21.723005 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:40:21.722988    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de03354dc312e857c09b247872ec83ad9ad0c2210108b7407bb2b66b82154a0\": container with ID starting with 1de03354dc312e857c09b247872ec83ad9ad0c2210108b7407bb2b66b82154a0 not found: ID does not exist" containerID="1de03354dc312e857c09b247872ec83ad9ad0c2210108b7407bb2b66b82154a0"
Jul 21 18:40:21.723059 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:21.723009    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de03354dc312e857c09b247872ec83ad9ad0c2210108b7407bb2b66b82154a0"} err="failed to get container status \"1de03354dc312e857c09b247872ec83ad9ad0c2210108b7407bb2b66b82154a0\": rpc error: code = NotFound desc = could not find container \"1de03354dc312e857c09b247872ec83ad9ad0c2210108b7407bb2b66b82154a0\": container with ID starting with 1de03354dc312e857c09b247872ec83ad9ad0c2210108b7407bb2b66b82154a0 not found: ID does not exist"
Jul 21 18:40:22.921219 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:40:22.921186    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc9d26a9-d89b-40a5-a1c8-4878691fc740" path="/var/lib/kubelet/pods/fc9d26a9-d89b-40a5-a1c8-4878691fc740/volumes"
Jul 21 18:41:35.760911 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:35.760877    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:41:35.763503 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:35.763479    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:41:45.362034 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.361981    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"]
Jul 21 18:41:45.362408 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.362297    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c5b3c61-b2af-4c46-bb1e-780008c20561" containerName="storage-initializer"
Jul 21 18:41:45.362408 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.362307    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c5b3c61-b2af-4c46-bb1e-780008c20561" containerName="storage-initializer"
Jul 21 18:41:45.362408 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.362317    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9d26a9-d89b-40a5-a1c8-4878691fc740" containerName="storage-initializer"
Jul 21 18:41:45.362408 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.362323    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9d26a9-d89b-40a5-a1c8-4878691fc740" containerName="storage-initializer"
Jul 21 18:41:45.362408 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.362333    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9d26a9-d89b-40a5-a1c8-4878691fc740" containerName="kserve-container"
Jul 21 18:41:45.362408 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.362339    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9d26a9-d89b-40a5-a1c8-4878691fc740" containerName="kserve-container"
Jul 21 18:41:45.362408 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.362347    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9d26a9-d89b-40a5-a1c8-4878691fc740" containerName="kube-rbac-proxy"
Jul 21 18:41:45.362408 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.362352    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9d26a9-d89b-40a5-a1c8-4878691fc740" containerName="kube-rbac-proxy"
Jul 21 18:41:45.362408 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.362360    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c5b3c61-b2af-4c46-bb1e-780008c20561" containerName="kube-rbac-proxy"
Jul 21 18:41:45.362408 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.362365    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c5b3c61-b2af-4c46-bb1e-780008c20561" containerName="kube-rbac-proxy"
Jul 21 18:41:45.362408 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.362376    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c5b3c61-b2af-4c46-bb1e-780008c20561" containerName="kserve-container"
Jul 21 18:41:45.362408 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.362380    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c5b3c61-b2af-4c46-bb1e-780008c20561" containerName="kserve-container"
Jul 21 18:41:45.362764 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.362431    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc9d26a9-d89b-40a5-a1c8-4878691fc740" containerName="kube-rbac-proxy"
Jul 21 18:41:45.362764 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.362439    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c5b3c61-b2af-4c46-bb1e-780008c20561" containerName="kube-rbac-proxy"
Jul 21 18:41:45.362764 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.362447    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc9d26a9-d89b-40a5-a1c8-4878691fc740" containerName="kserve-container"
Jul 21 18:41:45.362764 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.362454    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c5b3c61-b2af-4c46-bb1e-780008c20561" containerName="kserve-container"
Jul 21 18:41:45.365491 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.365474    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.368362 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.368338    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 21 18:41:45.368502 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.368401    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:41:45.369038 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.369022    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 21 18:41:45.369638 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.369620    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:41:45.369749 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.369638    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:41:45.369749 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.369672    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:41:45.369749 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.369690    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:41:45.378571 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.377998    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"]
Jul 21 18:41:45.469988 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.469958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcf2s\" (UniqueName: \"kubernetes.io/projected/32c488ba-0176-4dd9-9c29-972fefc49be4-kube-api-access-tcf2s\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.470135 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.469996    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32c488ba-0176-4dd9-9c29-972fefc49be4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.470135 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.470028    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32c488ba-0176-4dd9-9c29-972fefc49be4-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.470135 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.470097    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32c488ba-0176-4dd9-9c29-972fefc49be4-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.470246 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.470134    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32c488ba-0176-4dd9-9c29-972fefc49be4-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.570472 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.570439    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32c488ba-0176-4dd9-9c29-972fefc49be4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.570608 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.570484    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32c488ba-0176-4dd9-9c29-972fefc49be4-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.570608 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.570505    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32c488ba-0176-4dd9-9c29-972fefc49be4-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.570608 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.570534    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32c488ba-0176-4dd9-9c29-972fefc49be4-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.570608 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.570568    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcf2s\" (UniqueName: \"kubernetes.io/projected/32c488ba-0176-4dd9-9c29-972fefc49be4-kube-api-access-tcf2s\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.571031 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.571003    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32c488ba-0176-4dd9-9c29-972fefc49be4-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.571257 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.571237    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32c488ba-0176-4dd9-9c29-972fefc49be4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.571318 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.571239    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32c488ba-0176-4dd9-9c29-972fefc49be4-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.573086 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.573049    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32c488ba-0176-4dd9-9c29-972fefc49be4-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.578542 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.578520    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcf2s\" (UniqueName: \"kubernetes.io/projected/32c488ba-0176-4dd9-9c29-972fefc49be4-kube-api-access-tcf2s\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.676277 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.676255    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:45.805331 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:45.805303    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"]
Jul 21 18:41:45.807229 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:41:45.807204    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c488ba_0176_4dd9_9c29_972fefc49be4.slice/crio-b29ffbd5e3e5947f67628ca8fdc50e350723c95f59f0b5d2efe47ce0de5021a8 WatchSource:0}: Error finding container b29ffbd5e3e5947f67628ca8fdc50e350723c95f59f0b5d2efe47ce0de5021a8: Status 404 returned error can't find the container with id b29ffbd5e3e5947f67628ca8fdc50e350723c95f59f0b5d2efe47ce0de5021a8
Jul 21 18:41:46.023109 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:46.022999    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg" event={"ID":"32c488ba-0176-4dd9-9c29-972fefc49be4","Type":"ContainerStarted","Data":"50a96f5b4509280c85ae79ab7286710f229551b535e7a557b75d423c315742cd"}
Jul 21 18:41:46.023109 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:46.023105    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg" event={"ID":"32c488ba-0176-4dd9-9c29-972fefc49be4","Type":"ContainerStarted","Data":"b29ffbd5e3e5947f67628ca8fdc50e350723c95f59f0b5d2efe47ce0de5021a8"}
Jul 21 18:41:50.039575 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:50.039539    2562 generic.go:356] "Generic (PLEG): container finished" podID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerID="50a96f5b4509280c85ae79ab7286710f229551b535e7a557b75d423c315742cd" exitCode=0
Jul 21 18:41:50.039973 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:50.039587    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg" event={"ID":"32c488ba-0176-4dd9-9c29-972fefc49be4","Type":"ContainerDied","Data":"50a96f5b4509280c85ae79ab7286710f229551b535e7a557b75d423c315742cd"}
Jul 21 18:41:51.045931 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:51.045894    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg" event={"ID":"32c488ba-0176-4dd9-9c29-972fefc49be4","Type":"ContainerStarted","Data":"7c74a480d776f04bf395739507c870862b485e8e44eb5ac427d32bb51bdfd65c"}
Jul 21 18:41:51.046338 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:51.045958    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg" event={"ID":"32c488ba-0176-4dd9-9c29-972fefc49be4","Type":"ContainerStarted","Data":"698b726dd9a1f2b830dba3f1d202c7c0e259b60cc028d5c466377ed3a7b31bf4"}
Jul 21 18:41:51.046338 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:51.046199    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:51.046415 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:51.046375    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:41:51.067898 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:51.067855    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg" podStartSLOduration=2.537962092 podStartE2EDuration="6.067842081s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.529879989s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:41:51.066251711 +0000 UTC m=+4562.637873467" watchObservedRunningTime="2026-07-21 18:41:51.067842081 +0000 UTC m=+4562.639463835"
Jul 21 18:41:57.055767 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:41:57.055738    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:42:27.092585 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:27.092487    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg" podUID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 21 18:42:37.058647 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:37.058616    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:42:45.482227 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:45.482196    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"]
Jul 21 18:42:45.482644 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:45.482427    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg" podUID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerName="kserve-container" containerID="cri-o://698b726dd9a1f2b830dba3f1d202c7c0e259b60cc028d5c466377ed3a7b31bf4" gracePeriod=30
Jul 21 18:42:45.482644 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:45.482456    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg" podUID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerName="kube-rbac-proxy" containerID="cri-o://7c74a480d776f04bf395739507c870862b485e8e44eb5ac427d32bb51bdfd65c" gracePeriod=30
Jul 21 18:42:46.268610 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:46.268574    2562 generic.go:356] "Generic (PLEG): container finished" podID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerID="7c74a480d776f04bf395739507c870862b485e8e44eb5ac427d32bb51bdfd65c" exitCode=2
Jul 21 18:42:46.268775 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:46.268650    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg" event={"ID":"32c488ba-0176-4dd9-9c29-972fefc49be4","Type":"ContainerDied","Data":"7c74a480d776f04bf395739507c870862b485e8e44eb5ac427d32bb51bdfd65c"}
Jul 21 18:42:47.050764 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:47.050729    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg" podUID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Jul 21 18:42:48.097240 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:48.097186    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg" podUID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 21 18:42:52.050906 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.050862    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg" podUID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Jul 21 18:42:52.524012 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.523986    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:42:52.695704 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.695675    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32c488ba-0176-4dd9-9c29-972fefc49be4-kserve-provision-location\") pod \"32c488ba-0176-4dd9-9c29-972fefc49be4\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") "
Jul 21 18:42:52.695867 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.695745    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32c488ba-0176-4dd9-9c29-972fefc49be4-kube-rbac-proxy-sar-config\") pod \"32c488ba-0176-4dd9-9c29-972fefc49be4\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") "
Jul 21 18:42:52.695867 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.695770    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcf2s\" (UniqueName: \"kubernetes.io/projected/32c488ba-0176-4dd9-9c29-972fefc49be4-kube-api-access-tcf2s\") pod \"32c488ba-0176-4dd9-9c29-972fefc49be4\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") "
Jul 21 18:42:52.695867 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.695803    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32c488ba-0176-4dd9-9c29-972fefc49be4-proxy-tls\") pod \"32c488ba-0176-4dd9-9c29-972fefc49be4\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") "
Jul 21 18:42:52.695867 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.695839    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32c488ba-0176-4dd9-9c29-972fefc49be4-cabundle-cert\") pod \"32c488ba-0176-4dd9-9c29-972fefc49be4\" (UID: \"32c488ba-0176-4dd9-9c29-972fefc49be4\") "
Jul 21 18:42:52.696090 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.696028    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c488ba-0176-4dd9-9c29-972fefc49be4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32c488ba-0176-4dd9-9c29-972fefc49be4" (UID: "32c488ba-0176-4dd9-9c29-972fefc49be4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:42:52.696214 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.696120    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c488ba-0176-4dd9-9c29-972fefc49be4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "32c488ba-0176-4dd9-9c29-972fefc49be4" (UID: "32c488ba-0176-4dd9-9c29-972fefc49be4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:42:52.696326 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.696269    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c488ba-0176-4dd9-9c29-972fefc49be4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "32c488ba-0176-4dd9-9c29-972fefc49be4" (UID: "32c488ba-0176-4dd9-9c29-972fefc49be4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:42:52.698081 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.698044    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c488ba-0176-4dd9-9c29-972fefc49be4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32c488ba-0176-4dd9-9c29-972fefc49be4" (UID: "32c488ba-0176-4dd9-9c29-972fefc49be4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:42:52.698081 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.698052    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c488ba-0176-4dd9-9c29-972fefc49be4-kube-api-access-tcf2s" (OuterVolumeSpecName: "kube-api-access-tcf2s") pod "32c488ba-0176-4dd9-9c29-972fefc49be4" (UID: "32c488ba-0176-4dd9-9c29-972fefc49be4"). InnerVolumeSpecName "kube-api-access-tcf2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:42:52.796891 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.796848    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32c488ba-0176-4dd9-9c29-972fefc49be4-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:42:52.796891 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.796889    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32c488ba-0176-4dd9-9c29-972fefc49be4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:42:52.796891 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.796902    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcf2s\" (UniqueName: \"kubernetes.io/projected/32c488ba-0176-4dd9-9c29-972fefc49be4-kube-api-access-tcf2s\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:42:52.797148 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.796916    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32c488ba-0176-4dd9-9c29-972fefc49be4-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:42:52.797148 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:52.796928    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32c488ba-0176-4dd9-9c29-972fefc49be4-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:42:53.296847 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:53.296809    2562 generic.go:356] "Generic (PLEG): container finished" podID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerID="698b726dd9a1f2b830dba3f1d202c7c0e259b60cc028d5c466377ed3a7b31bf4" exitCode=0
Jul 21 18:42:53.297357 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:53.296863    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"
Jul 21 18:42:53.297357 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:53.296886    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg" event={"ID":"32c488ba-0176-4dd9-9c29-972fefc49be4","Type":"ContainerDied","Data":"698b726dd9a1f2b830dba3f1d202c7c0e259b60cc028d5c466377ed3a7b31bf4"}
Jul 21 18:42:53.297357 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:53.296948    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg" event={"ID":"32c488ba-0176-4dd9-9c29-972fefc49be4","Type":"ContainerDied","Data":"b29ffbd5e3e5947f67628ca8fdc50e350723c95f59f0b5d2efe47ce0de5021a8"}
Jul 21 18:42:53.297357 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:53.296978    2562 scope.go:117] "RemoveContainer" containerID="7c74a480d776f04bf395739507c870862b485e8e44eb5ac427d32bb51bdfd65c"
Jul 21 18:42:53.305828 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:53.305809    2562 scope.go:117] "RemoveContainer" containerID="698b726dd9a1f2b830dba3f1d202c7c0e259b60cc028d5c466377ed3a7b31bf4"
Jul 21 18:42:53.316032 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:53.316006    2562 scope.go:117] "RemoveContainer" containerID="50a96f5b4509280c85ae79ab7286710f229551b535e7a557b75d423c315742cd"
Jul 21 18:42:53.317962 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:53.317942    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"]
Jul 21 18:42:53.320603 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:53.320583    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-xv7vg"]
Jul 21 18:42:53.324193 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:53.324179    2562 scope.go:117] "RemoveContainer" containerID="7c74a480d776f04bf395739507c870862b485e8e44eb5ac427d32bb51bdfd65c"
Jul 21 18:42:53.324458 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:42:53.324441    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c74a480d776f04bf395739507c870862b485e8e44eb5ac427d32bb51bdfd65c\": container with ID starting with 7c74a480d776f04bf395739507c870862b485e8e44eb5ac427d32bb51bdfd65c not found: ID does not exist" containerID="7c74a480d776f04bf395739507c870862b485e8e44eb5ac427d32bb51bdfd65c"
Jul 21 18:42:53.324518 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:53.324472    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c74a480d776f04bf395739507c870862b485e8e44eb5ac427d32bb51bdfd65c"} err="failed to get container status \"7c74a480d776f04bf395739507c870862b485e8e44eb5ac427d32bb51bdfd65c\": rpc error: code = NotFound desc = could not find container \"7c74a480d776f04bf395739507c870862b485e8e44eb5ac427d32bb51bdfd65c\": container with ID starting with 7c74a480d776f04bf395739507c870862b485e8e44eb5ac427d32bb51bdfd65c not found: ID does not exist"
Jul 21 18:42:53.324518 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:53.324488    2562 scope.go:117] "RemoveContainer" containerID="698b726dd9a1f2b830dba3f1d202c7c0e259b60cc028d5c466377ed3a7b31bf4"
Jul 21 18:42:53.324741 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:42:53.324717    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698b726dd9a1f2b830dba3f1d202c7c0e259b60cc028d5c466377ed3a7b31bf4\": container with ID starting with 698b726dd9a1f2b830dba3f1d202c7c0e259b60cc028d5c466377ed3a7b31bf4 not found: ID does not exist" containerID="698b726dd9a1f2b830dba3f1d202c7c0e259b60cc028d5c466377ed3a7b31bf4"
Jul 21 18:42:53.324800 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:53.324747    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698b726dd9a1f2b830dba3f1d202c7c0e259b60cc028d5c466377ed3a7b31bf4"} err="failed to get container status \"698b726dd9a1f2b830dba3f1d202c7c0e259b60cc028d5c466377ed3a7b31bf4\": rpc error: code = NotFound desc = could not find container \"698b726dd9a1f2b830dba3f1d202c7c0e259b60cc028d5c466377ed3a7b31bf4\": container with ID starting with 698b726dd9a1f2b830dba3f1d202c7c0e259b60cc028d5c466377ed3a7b31bf4 not found: ID does not exist"
Jul 21 18:42:53.324800 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:53.324762    2562 scope.go:117] "RemoveContainer" containerID="50a96f5b4509280c85ae79ab7286710f229551b535e7a557b75d423c315742cd"
Jul 21 18:42:53.324986 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:42:53.324964    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a96f5b4509280c85ae79ab7286710f229551b535e7a557b75d423c315742cd\": container with ID starting with 50a96f5b4509280c85ae79ab7286710f229551b535e7a557b75d423c315742cd not found: ID does not exist" containerID="50a96f5b4509280c85ae79ab7286710f229551b535e7a557b75d423c315742cd"
Jul 21 18:42:53.325040 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:53.324994    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a96f5b4509280c85ae79ab7286710f229551b535e7a557b75d423c315742cd"} err="failed to get container status \"50a96f5b4509280c85ae79ab7286710f229551b535e7a557b75d423c315742cd\": rpc error: code = NotFound desc = could not find container \"50a96f5b4509280c85ae79ab7286710f229551b535e7a557b75d423c315742cd\": container with ID starting with 50a96f5b4509280c85ae79ab7286710f229551b535e7a557b75d423c315742cd not found: ID does not exist"
Jul 21 18:42:54.921209 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:42:54.921178    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32c488ba-0176-4dd9-9c29-972fefc49be4" path="/var/lib/kubelet/pods/32c488ba-0176-4dd9-9c29-972fefc49be4/volumes"
Jul 21 18:44:05.945025 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.944988    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"]
Jul 21 18:44:05.945473 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.945311    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerName="kserve-container"
Jul 21 18:44:05.945473 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.945322    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerName="kserve-container"
Jul 21 18:44:05.945473 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.945335    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerName="kube-rbac-proxy"
Jul 21 18:44:05.945473 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.945342    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerName="kube-rbac-proxy"
Jul 21 18:44:05.945473 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.945355    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerName="storage-initializer"
Jul 21 18:44:05.945473 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.945360    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerName="storage-initializer"
Jul 21 18:44:05.945473 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.945408    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerName="kserve-container"
Jul 21 18:44:05.945473 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.945415    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c488ba-0176-4dd9-9c29-972fefc49be4" containerName="kube-rbac-proxy"
Jul 21 18:44:05.949878 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.949856    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:05.952493 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.952472    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jul 21 18:44:05.952683 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.952660    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:44:05.952801 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.952547    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 21 18:44:05.952801 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.952567    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jul 21 18:44:05.952801 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.952755    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:44:05.952980 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.952616    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:44:05.959846 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:05.959827    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"]
Jul 21 18:44:06.048832 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:06.048803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1094d4e-d58d-4283-a766-35605acfdd76-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6b8556897c-578q9\" (UID: \"b1094d4e-d58d-4283-a766-35605acfdd76\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:06.048969 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:06.048869    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xvk\" (UniqueName: \"kubernetes.io/projected/b1094d4e-d58d-4283-a766-35605acfdd76-kube-api-access-s5xvk\") pod \"isvc-sklearn-s3-predictor-6b8556897c-578q9\" (UID: \"b1094d4e-d58d-4283-a766-35605acfdd76\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:06.048969 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:06.048931    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1094d4e-d58d-4283-a766-35605acfdd76-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6b8556897c-578q9\" (UID: \"b1094d4e-d58d-4283-a766-35605acfdd76\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:06.048969 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:06.048958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1094d4e-d58d-4283-a766-35605acfdd76-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6b8556897c-578q9\" (UID: \"b1094d4e-d58d-4283-a766-35605acfdd76\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:06.149422 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:06.149390    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1094d4e-d58d-4283-a766-35605acfdd76-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6b8556897c-578q9\" (UID: \"b1094d4e-d58d-4283-a766-35605acfdd76\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:06.149422 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:06.149422    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1094d4e-d58d-4283-a766-35605acfdd76-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6b8556897c-578q9\" (UID: \"b1094d4e-d58d-4283-a766-35605acfdd76\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:06.149657 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:06.149442    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1094d4e-d58d-4283-a766-35605acfdd76-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6b8556897c-578q9\" (UID: \"b1094d4e-d58d-4283-a766-35605acfdd76\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:06.149657 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:06.149484    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xvk\" (UniqueName: \"kubernetes.io/projected/b1094d4e-d58d-4283-a766-35605acfdd76-kube-api-access-s5xvk\") pod \"isvc-sklearn-s3-predictor-6b8556897c-578q9\" (UID: \"b1094d4e-d58d-4283-a766-35605acfdd76\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:06.149874 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:06.149854    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1094d4e-d58d-4283-a766-35605acfdd76-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6b8556897c-578q9\" (UID: \"b1094d4e-d58d-4283-a766-35605acfdd76\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:06.150198 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:06.150181    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1094d4e-d58d-4283-a766-35605acfdd76-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6b8556897c-578q9\" (UID: \"b1094d4e-d58d-4283-a766-35605acfdd76\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:06.151983 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:06.151963    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1094d4e-d58d-4283-a766-35605acfdd76-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6b8556897c-578q9\" (UID: \"b1094d4e-d58d-4283-a766-35605acfdd76\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:06.158289 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:06.158265    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xvk\" (UniqueName: \"kubernetes.io/projected/b1094d4e-d58d-4283-a766-35605acfdd76-kube-api-access-s5xvk\") pod \"isvc-sklearn-s3-predictor-6b8556897c-578q9\" (UID: \"b1094d4e-d58d-4283-a766-35605acfdd76\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:06.261526 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:06.261449    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:06.395558 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:06.395535    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"]
Jul 21 18:44:06.398175 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:44:06.398129    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1094d4e_d58d_4283_a766_35605acfdd76.slice/crio-aee3bf32e9af07c0d125c73d6dc5aa4ebf87c7cbfc0be1919fbc4758a3798536 WatchSource:0}: Error finding container aee3bf32e9af07c0d125c73d6dc5aa4ebf87c7cbfc0be1919fbc4758a3798536: Status 404 returned error can't find the container with id aee3bf32e9af07c0d125c73d6dc5aa4ebf87c7cbfc0be1919fbc4758a3798536
Jul 21 18:44:06.578164 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:06.578056    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" event={"ID":"b1094d4e-d58d-4283-a766-35605acfdd76","Type":"ContainerStarted","Data":"1a7056d326918a72df70f710fae35d1d0c3bb649cc09ac4fe635a2d9c17527ea"}
Jul 21 18:44:06.578164 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:06.578156    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" event={"ID":"b1094d4e-d58d-4283-a766-35605acfdd76","Type":"ContainerStarted","Data":"aee3bf32e9af07c0d125c73d6dc5aa4ebf87c7cbfc0be1919fbc4758a3798536"}
Jul 21 18:44:07.583526 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:07.583488    2562 generic.go:356] "Generic (PLEG): container finished" podID="b1094d4e-d58d-4283-a766-35605acfdd76" containerID="1a7056d326918a72df70f710fae35d1d0c3bb649cc09ac4fe635a2d9c17527ea" exitCode=0
Jul 21 18:44:07.583977 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:07.583577    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" event={"ID":"b1094d4e-d58d-4283-a766-35605acfdd76","Type":"ContainerDied","Data":"1a7056d326918a72df70f710fae35d1d0c3bb649cc09ac4fe635a2d9c17527ea"}
Jul 21 18:44:08.589267 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:08.589234    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" event={"ID":"b1094d4e-d58d-4283-a766-35605acfdd76","Type":"ContainerStarted","Data":"de419341c145b35ec4c7edb84aec13b0e568a489905756cb63a53416ed443fac"}
Jul 21 18:44:08.589643 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:08.589292    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" event={"ID":"b1094d4e-d58d-4283-a766-35605acfdd76","Type":"ContainerStarted","Data":"8c37681666460c56b03215c6f816de1d160e1d1f7506009841a0a747904fd104"}
Jul 21 18:44:08.589643 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:08.589366    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:08.608773 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:08.608724    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" podStartSLOduration=2.647474189 podStartE2EDuration="3.608709537s" totalImagesPullingTime="0s" totalInitContainerRuntime="961.235348ms" isStatefulPod=false podCreationTimestamp="2026-07-21 18:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:44:08.607377457 +0000 UTC m=+4700.178999212" watchObservedRunningTime="2026-07-21 18:44:08.608709537 +0000 UTC m=+4700.180331292"
Jul 21 18:44:09.593031 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:09.592989    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:09.594496 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:09.594471    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 21 18:44:10.597956 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:10.597918    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 21 18:44:15.603197 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:15.603168    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:44:15.603718 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:15.603641    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 21 18:44:25.603732 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:25.603691    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 21 18:44:35.603606 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:35.603569    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 21 18:44:45.603601 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:45.603555    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 21 18:44:55.604373 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:44:55.604306    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 21 18:45:05.604547 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:05.604509    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 21 18:45:15.604454 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:15.604373    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 21 18:45:25.604667 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:25.604638    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:45:26.064158 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.061770    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"]
Jul 21 18:45:26.065388 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.065340    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kserve-container" containerID="cri-o://8c37681666460c56b03215c6f816de1d160e1d1f7506009841a0a747904fd104" gracePeriod=30
Jul 21 18:45:26.065791 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.065761    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kube-rbac-proxy" containerID="cri-o://de419341c145b35ec4c7edb84aec13b0e568a489905756cb63a53416ed443fac" gracePeriod=30
Jul 21 18:45:26.214714 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.214685    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"]
Jul 21 18:45:26.218012 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.217992    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.220553 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.220532    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 21 18:45:26.220707 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.220572    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 21 18:45:26.220803 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.220783    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:45:26.227656 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.227629    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"]
Jul 21 18:45:26.391562 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.391479    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b59894f-5aac-434f-a49e-5da3b179f6f0-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.391562 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.391523    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b59894f-5aac-434f-a49e-5da3b179f6f0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.391562 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.391556    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b59894f-5aac-434f-a49e-5da3b179f6f0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.391805 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.391581    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khrdx\" (UniqueName: \"kubernetes.io/projected/0b59894f-5aac-434f-a49e-5da3b179f6f0-kube-api-access-khrdx\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.391805 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.391684    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b59894f-5aac-434f-a49e-5da3b179f6f0-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.492944 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.492911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b59894f-5aac-434f-a49e-5da3b179f6f0-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.493136 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.492955    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b59894f-5aac-434f-a49e-5da3b179f6f0-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.493136 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.492988    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b59894f-5aac-434f-a49e-5da3b179f6f0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.493136 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.493016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b59894f-5aac-434f-a49e-5da3b179f6f0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.493136 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.493042    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khrdx\" (UniqueName: \"kubernetes.io/projected/0b59894f-5aac-434f-a49e-5da3b179f6f0-kube-api-access-khrdx\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.493518 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.493493    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b59894f-5aac-434f-a49e-5da3b179f6f0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.493676 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.493659    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b59894f-5aac-434f-a49e-5da3b179f6f0-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.493805 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.493782    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b59894f-5aac-434f-a49e-5da3b179f6f0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.495614 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.495596    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b59894f-5aac-434f-a49e-5da3b179f6f0-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.501151 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.501126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khrdx\" (UniqueName: \"kubernetes.io/projected/0b59894f-5aac-434f-a49e-5da3b179f6f0-kube-api-access-khrdx\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.528733 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.528710    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:26.662956 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.662926    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"]
Jul 21 18:45:26.666136 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:45:26.666116    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b59894f_5aac_434f_a49e_5da3b179f6f0.slice/crio-6e99004fdbed4f30757e6d7e0281bfb928c0f4a3097f683318058762b9c17ef6 WatchSource:0}: Error finding container 6e99004fdbed4f30757e6d7e0281bfb928c0f4a3097f683318058762b9c17ef6: Status 404 returned error can't find the container with id 6e99004fdbed4f30757e6d7e0281bfb928c0f4a3097f683318058762b9c17ef6
Jul 21 18:45:26.912196 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.912166    2562 generic.go:356] "Generic (PLEG): container finished" podID="b1094d4e-d58d-4283-a766-35605acfdd76" containerID="de419341c145b35ec4c7edb84aec13b0e568a489905756cb63a53416ed443fac" exitCode=2
Jul 21 18:45:26.912511 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.912246    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" event={"ID":"b1094d4e-d58d-4283-a766-35605acfdd76","Type":"ContainerDied","Data":"de419341c145b35ec4c7edb84aec13b0e568a489905756cb63a53416ed443fac"}
Jul 21 18:45:26.913964 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.913899    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" event={"ID":"0b59894f-5aac-434f-a49e-5da3b179f6f0","Type":"ContainerStarted","Data":"c8bcd0cb2d39e260304d3b28f46da4d90d5e562e65e6d7cb5440d7fbbc2e6288"}
Jul 21 18:45:26.913964 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:26.913951    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" event={"ID":"0b59894f-5aac-434f-a49e-5da3b179f6f0","Type":"ContainerStarted","Data":"6e99004fdbed4f30757e6d7e0281bfb928c0f4a3097f683318058762b9c17ef6"}
Jul 21 18:45:27.919167 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:27.919083    2562 generic.go:356] "Generic (PLEG): container finished" podID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerID="c8bcd0cb2d39e260304d3b28f46da4d90d5e562e65e6d7cb5440d7fbbc2e6288" exitCode=0
Jul 21 18:45:27.919543 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:27.919158    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" event={"ID":"0b59894f-5aac-434f-a49e-5da3b179f6f0","Type":"ContainerDied","Data":"c8bcd0cb2d39e260304d3b28f46da4d90d5e562e65e6d7cb5440d7fbbc2e6288"}
Jul 21 18:45:28.925125 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:28.925094    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" event={"ID":"0b59894f-5aac-434f-a49e-5da3b179f6f0","Type":"ContainerStarted","Data":"902b72c6c517316489e43937189354a50184421898f1ea5da2b61b3183a7b9fa"}
Jul 21 18:45:28.925525 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:28.925147    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" event={"ID":"0b59894f-5aac-434f-a49e-5da3b179f6f0","Type":"ContainerStarted","Data":"a5aff21265bf204040fe4ed18385692cf629a0f6a313003a36fed60b432be498"}
Jul 21 18:45:28.925525 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:28.925180    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:28.947855 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:28.947810    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" podStartSLOduration=2.009938002 podStartE2EDuration="2.947794959s" totalImagesPullingTime="0s" totalInitContainerRuntime="937.856957ms" isStatefulPod=false podCreationTimestamp="2026-07-21 18:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:45:28.945984978 +0000 UTC m=+4780.517606745" watchObservedRunningTime="2026-07-21 18:45:28.947794959 +0000 UTC m=+4780.519416714"
Jul 21 18:45:29.930038 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:29.930009    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:29.931400 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:29.931372    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 21 18:45:30.611839 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.611816    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:45:30.731782 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.731753    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1094d4e-d58d-4283-a766-35605acfdd76-kserve-provision-location\") pod \"b1094d4e-d58d-4283-a766-35605acfdd76\" (UID: \"b1094d4e-d58d-4283-a766-35605acfdd76\") "
Jul 21 18:45:30.731917 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.731802    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1094d4e-d58d-4283-a766-35605acfdd76-kube-rbac-proxy-sar-config\") pod \"b1094d4e-d58d-4283-a766-35605acfdd76\" (UID: \"b1094d4e-d58d-4283-a766-35605acfdd76\") "
Jul 21 18:45:30.731917 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.731893    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1094d4e-d58d-4283-a766-35605acfdd76-proxy-tls\") pod \"b1094d4e-d58d-4283-a766-35605acfdd76\" (UID: \"b1094d4e-d58d-4283-a766-35605acfdd76\") "
Jul 21 18:45:30.731917 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.731912    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5xvk\" (UniqueName: \"kubernetes.io/projected/b1094d4e-d58d-4283-a766-35605acfdd76-kube-api-access-s5xvk\") pod \"b1094d4e-d58d-4283-a766-35605acfdd76\" (UID: \"b1094d4e-d58d-4283-a766-35605acfdd76\") "
Jul 21 18:45:30.732132 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.732090    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1094d4e-d58d-4283-a766-35605acfdd76-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1094d4e-d58d-4283-a766-35605acfdd76" (UID: "b1094d4e-d58d-4283-a766-35605acfdd76"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:45:30.732208 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.732183    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1094d4e-d58d-4283-a766-35605acfdd76-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1094d4e-d58d-4283-a766-35605acfdd76" (UID: "b1094d4e-d58d-4283-a766-35605acfdd76"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:45:30.734104 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.734056    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1094d4e-d58d-4283-a766-35605acfdd76-kube-api-access-s5xvk" (OuterVolumeSpecName: "kube-api-access-s5xvk") pod "b1094d4e-d58d-4283-a766-35605acfdd76" (UID: "b1094d4e-d58d-4283-a766-35605acfdd76"). InnerVolumeSpecName "kube-api-access-s5xvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:45:30.734297 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.734281    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1094d4e-d58d-4283-a766-35605acfdd76-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1094d4e-d58d-4283-a766-35605acfdd76" (UID: "b1094d4e-d58d-4283-a766-35605acfdd76"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:45:30.832908 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.832876    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1094d4e-d58d-4283-a766-35605acfdd76-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:45:30.832908 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.832901    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1094d4e-d58d-4283-a766-35605acfdd76-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:45:30.832908 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.832910    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5xvk\" (UniqueName: \"kubernetes.io/projected/b1094d4e-d58d-4283-a766-35605acfdd76-kube-api-access-s5xvk\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:45:30.833154 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.832920    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1094d4e-d58d-4283-a766-35605acfdd76-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:45:30.935711 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.935679    2562 generic.go:356] "Generic (PLEG): container finished" podID="b1094d4e-d58d-4283-a766-35605acfdd76" containerID="8c37681666460c56b03215c6f816de1d160e1d1f7506009841a0a747904fd104" exitCode=0
Jul 21 18:45:30.936085 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.935731    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"
Jul 21 18:45:30.936085 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.935744    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" event={"ID":"b1094d4e-d58d-4283-a766-35605acfdd76","Type":"ContainerDied","Data":"8c37681666460c56b03215c6f816de1d160e1d1f7506009841a0a747904fd104"}
Jul 21 18:45:30.936085 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.935798    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" event={"ID":"b1094d4e-d58d-4283-a766-35605acfdd76","Type":"ContainerDied","Data":"aee3bf32e9af07c0d125c73d6dc5aa4ebf87c7cbfc0be1919fbc4758a3798536"}
Jul 21 18:45:30.936085 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.935815    2562 scope.go:117] "RemoveContainer" containerID="de419341c145b35ec4c7edb84aec13b0e568a489905756cb63a53416ed443fac"
Jul 21 18:45:30.936403 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.936377    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 21 18:45:30.944038 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.944022    2562 scope.go:117] "RemoveContainer" containerID="8c37681666460c56b03215c6f816de1d160e1d1f7506009841a0a747904fd104"
Jul 21 18:45:30.951735 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.951691    2562 scope.go:117] "RemoveContainer" containerID="1a7056d326918a72df70f710fae35d1d0c3bb649cc09ac4fe635a2d9c17527ea"
Jul 21 18:45:30.953240 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.953219    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"]
Jul 21 18:45:30.956743 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.956726    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9"]
Jul 21 18:45:30.959603 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.959590    2562 scope.go:117] "RemoveContainer" containerID="de419341c145b35ec4c7edb84aec13b0e568a489905756cb63a53416ed443fac"
Jul 21 18:45:30.959841 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:45:30.959825    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de419341c145b35ec4c7edb84aec13b0e568a489905756cb63a53416ed443fac\": container with ID starting with de419341c145b35ec4c7edb84aec13b0e568a489905756cb63a53416ed443fac not found: ID does not exist" containerID="de419341c145b35ec4c7edb84aec13b0e568a489905756cb63a53416ed443fac"
Jul 21 18:45:30.959893 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.959848    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de419341c145b35ec4c7edb84aec13b0e568a489905756cb63a53416ed443fac"} err="failed to get container status \"de419341c145b35ec4c7edb84aec13b0e568a489905756cb63a53416ed443fac\": rpc error: code = NotFound desc = could not find container \"de419341c145b35ec4c7edb84aec13b0e568a489905756cb63a53416ed443fac\": container with ID starting with de419341c145b35ec4c7edb84aec13b0e568a489905756cb63a53416ed443fac not found: ID does not exist"
Jul 21 18:45:30.959893 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.959867    2562 scope.go:117] "RemoveContainer" containerID="8c37681666460c56b03215c6f816de1d160e1d1f7506009841a0a747904fd104"
Jul 21 18:45:30.960106 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:45:30.960082    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c37681666460c56b03215c6f816de1d160e1d1f7506009841a0a747904fd104\": container with ID starting with 8c37681666460c56b03215c6f816de1d160e1d1f7506009841a0a747904fd104 not found: ID does not exist" containerID="8c37681666460c56b03215c6f816de1d160e1d1f7506009841a0a747904fd104"
Jul 21 18:45:30.960182 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.960108    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c37681666460c56b03215c6f816de1d160e1d1f7506009841a0a747904fd104"} err="failed to get container status \"8c37681666460c56b03215c6f816de1d160e1d1f7506009841a0a747904fd104\": rpc error: code = NotFound desc = could not find container \"8c37681666460c56b03215c6f816de1d160e1d1f7506009841a0a747904fd104\": container with ID starting with 8c37681666460c56b03215c6f816de1d160e1d1f7506009841a0a747904fd104 not found: ID does not exist"
Jul 21 18:45:30.960182 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.960122    2562 scope.go:117] "RemoveContainer" containerID="1a7056d326918a72df70f710fae35d1d0c3bb649cc09ac4fe635a2d9c17527ea"
Jul 21 18:45:30.960379 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:45:30.960362    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a7056d326918a72df70f710fae35d1d0c3bb649cc09ac4fe635a2d9c17527ea\": container with ID starting with 1a7056d326918a72df70f710fae35d1d0c3bb649cc09ac4fe635a2d9c17527ea not found: ID does not exist" containerID="1a7056d326918a72df70f710fae35d1d0c3bb649cc09ac4fe635a2d9c17527ea"
Jul 21 18:45:30.960434 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:30.960389    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a7056d326918a72df70f710fae35d1d0c3bb649cc09ac4fe635a2d9c17527ea"} err="failed to get container status \"1a7056d326918a72df70f710fae35d1d0c3bb649cc09ac4fe635a2d9c17527ea\": rpc error: code = NotFound desc = could not find container \"1a7056d326918a72df70f710fae35d1d0c3bb649cc09ac4fe635a2d9c17527ea\": container with ID starting with 1a7056d326918a72df70f710fae35d1d0c3bb649cc09ac4fe635a2d9c17527ea not found: ID does not exist"
Jul 21 18:45:31.599225 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:31.599187    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b8556897c-578q9" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": context deadline exceeded"
Jul 21 18:45:32.920896 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:32.920863    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" path="/var/lib/kubelet/pods/b1094d4e-d58d-4283-a766-35605acfdd76/volumes"
Jul 21 18:45:35.940734 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:35.940711    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:45:35.941301 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:35.941276    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 21 18:45:45.941698 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:45.941660    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 21 18:45:55.941481 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:45:55.941436    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 21 18:46:05.941177 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:05.941139    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 21 18:46:15.941351 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:15.941313    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 21 18:46:25.941707 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:25.941661    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 21 18:46:35.789853 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:35.789826    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:46:35.793560 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:35.793540    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:46:35.942020 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:35.941984    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 21 18:46:41.917285 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:41.917250    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:46:46.285189 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:46.285153    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"]
Jul 21 18:46:46.285563 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:46.285383    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kserve-container" containerID="cri-o://a5aff21265bf204040fe4ed18385692cf629a0f6a313003a36fed60b432be498" gracePeriod=30
Jul 21 18:46:46.285563 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:46.285429    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kube-rbac-proxy" containerID="cri-o://902b72c6c517316489e43937189354a50184421898f1ea5da2b61b3183a7b9fa" gracePeriod=30
Jul 21 18:46:47.243641 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.243604    2562 generic.go:356] "Generic (PLEG): container finished" podID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerID="902b72c6c517316489e43937189354a50184421898f1ea5da2b61b3183a7b9fa" exitCode=2
Jul 21 18:46:47.243809 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.243675    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" event={"ID":"0b59894f-5aac-434f-a49e-5da3b179f6f0","Type":"ContainerDied","Data":"902b72c6c517316489e43937189354a50184421898f1ea5da2b61b3183a7b9fa"}
Jul 21 18:46:47.360042 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.360012    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"]
Jul 21 18:46:47.360459 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.360348    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kube-rbac-proxy"
Jul 21 18:46:47.360459 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.360359    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kube-rbac-proxy"
Jul 21 18:46:47.360459 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.360371    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kserve-container"
Jul 21 18:46:47.360459 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.360376    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kserve-container"
Jul 21 18:46:47.360459 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.360387    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="storage-initializer"
Jul 21 18:46:47.360459 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.360393    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="storage-initializer"
Jul 21 18:46:47.360459 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.360443    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kserve-container"
Jul 21 18:46:47.360459 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.360460    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1094d4e-d58d-4283-a766-35605acfdd76" containerName="kube-rbac-proxy"
Jul 21 18:46:47.363480 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.363462    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"
Jul 21 18:46:47.365764 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.365747    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jul 21 18:46:47.365961 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.365947    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jul 21 18:46:47.374503 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.374482    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"]
Jul 21 18:46:47.522218 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.522127    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3348dec-ec30-4f66-b710-d8e19ea037e6-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b\" (UID: \"c3348dec-ec30-4f66-b710-d8e19ea037e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"
Jul 21 18:46:47.522218 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.522184    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8pnk\" (UniqueName: \"kubernetes.io/projected/c3348dec-ec30-4f66-b710-d8e19ea037e6-kube-api-access-r8pnk\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b\" (UID: \"c3348dec-ec30-4f66-b710-d8e19ea037e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"
Jul 21 18:46:47.522218 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.522219    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3348dec-ec30-4f66-b710-d8e19ea037e6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b\" (UID: \"c3348dec-ec30-4f66-b710-d8e19ea037e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"
Jul 21 18:46:47.522432 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.522241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3348dec-ec30-4f66-b710-d8e19ea037e6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b\" (UID: \"c3348dec-ec30-4f66-b710-d8e19ea037e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"
Jul 21 18:46:47.623485 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.623453    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3348dec-ec30-4f66-b710-d8e19ea037e6-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b\" (UID: \"c3348dec-ec30-4f66-b710-d8e19ea037e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"
Jul 21 18:46:47.623649 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.623507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8pnk\" (UniqueName: \"kubernetes.io/projected/c3348dec-ec30-4f66-b710-d8e19ea037e6-kube-api-access-r8pnk\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b\" (UID: \"c3348dec-ec30-4f66-b710-d8e19ea037e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"
Jul 21 18:46:47.623649 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.623526    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3348dec-ec30-4f66-b710-d8e19ea037e6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b\" (UID: \"c3348dec-ec30-4f66-b710-d8e19ea037e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"
Jul 21 18:46:47.623649 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.623549    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3348dec-ec30-4f66-b710-d8e19ea037e6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b\" (UID: \"c3348dec-ec30-4f66-b710-d8e19ea037e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"
Jul 21 18:46:47.623941 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.623925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3348dec-ec30-4f66-b710-d8e19ea037e6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b\" (UID: \"c3348dec-ec30-4f66-b710-d8e19ea037e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"
Jul 21 18:46:47.624262 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.624246    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3348dec-ec30-4f66-b710-d8e19ea037e6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b\" (UID: \"c3348dec-ec30-4f66-b710-d8e19ea037e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"
Jul 21 18:46:47.626197 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.626178    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3348dec-ec30-4f66-b710-d8e19ea037e6-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b\" (UID: \"c3348dec-ec30-4f66-b710-d8e19ea037e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"
Jul 21 18:46:47.631101 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.631083    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8pnk\" (UniqueName: \"kubernetes.io/projected/c3348dec-ec30-4f66-b710-d8e19ea037e6-kube-api-access-r8pnk\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b\" (UID: \"c3348dec-ec30-4f66-b710-d8e19ea037e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"
Jul 21 18:46:47.673708 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.673685    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"
Jul 21 18:46:47.804098 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:47.804059    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"]
Jul 21 18:46:47.806860 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:46:47.806826    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3348dec_ec30_4f66_b710_d8e19ea037e6.slice/crio-ee31cfc3259a24c3fdd88727a2412131c6b83a47166e6d1f94ce80a76624a012 WatchSource:0}: Error finding container ee31cfc3259a24c3fdd88727a2412131c6b83a47166e6d1f94ce80a76624a012: Status 404 returned error can't find the container with id ee31cfc3259a24c3fdd88727a2412131c6b83a47166e6d1f94ce80a76624a012
Jul 21 18:46:48.251455 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:48.251424    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b" event={"ID":"c3348dec-ec30-4f66-b710-d8e19ea037e6","Type":"ContainerStarted","Data":"0496925978b5cec05a4ce231cb8c4814e1c1a7844ee6bf8282b69de51fff02f2"}
Jul 21 18:46:48.251639 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:48.251479    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b" event={"ID":"c3348dec-ec30-4f66-b710-d8e19ea037e6","Type":"ContainerStarted","Data":"ee31cfc3259a24c3fdd88727a2412131c6b83a47166e6d1f94ce80a76624a012"}
Jul 21 18:46:50.834110 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:50.834088    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:46:50.950297 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:50.950264    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b59894f-5aac-434f-a49e-5da3b179f6f0-proxy-tls\") pod \"0b59894f-5aac-434f-a49e-5da3b179f6f0\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") "
Jul 21 18:46:50.950461 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:50.950309    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b59894f-5aac-434f-a49e-5da3b179f6f0-kserve-provision-location\") pod \"0b59894f-5aac-434f-a49e-5da3b179f6f0\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") "
Jul 21 18:46:50.950461 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:50.950362    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b59894f-5aac-434f-a49e-5da3b179f6f0-cabundle-cert\") pod \"0b59894f-5aac-434f-a49e-5da3b179f6f0\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") "
Jul 21 18:46:50.950461 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:50.950417    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b59894f-5aac-434f-a49e-5da3b179f6f0-kube-rbac-proxy-sar-config\") pod \"0b59894f-5aac-434f-a49e-5da3b179f6f0\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") "
Jul 21 18:46:50.950461 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:50.950440    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khrdx\" (UniqueName: \"kubernetes.io/projected/0b59894f-5aac-434f-a49e-5da3b179f6f0-kube-api-access-khrdx\") pod \"0b59894f-5aac-434f-a49e-5da3b179f6f0\" (UID: \"0b59894f-5aac-434f-a49e-5da3b179f6f0\") "
Jul 21 18:46:50.950757 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:50.950690    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b59894f-5aac-434f-a49e-5da3b179f6f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b59894f-5aac-434f-a49e-5da3b179f6f0" (UID: "0b59894f-5aac-434f-a49e-5da3b179f6f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:46:50.950757 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:50.950712    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b59894f-5aac-434f-a49e-5da3b179f6f0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b59894f-5aac-434f-a49e-5da3b179f6f0" (UID: "0b59894f-5aac-434f-a49e-5da3b179f6f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:46:50.950757 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:50.950726    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b59894f-5aac-434f-a49e-5da3b179f6f0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b59894f-5aac-434f-a49e-5da3b179f6f0" (UID: "0b59894f-5aac-434f-a49e-5da3b179f6f0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:46:50.952664 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:50.952641    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b59894f-5aac-434f-a49e-5da3b179f6f0-kube-api-access-khrdx" (OuterVolumeSpecName: "kube-api-access-khrdx") pod "0b59894f-5aac-434f-a49e-5da3b179f6f0" (UID: "0b59894f-5aac-434f-a49e-5da3b179f6f0"). InnerVolumeSpecName "kube-api-access-khrdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:46:50.952664 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:50.952653    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b59894f-5aac-434f-a49e-5da3b179f6f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b59894f-5aac-434f-a49e-5da3b179f6f0" (UID: "0b59894f-5aac-434f-a49e-5da3b179f6f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:46:51.051258 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.051232    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b59894f-5aac-434f-a49e-5da3b179f6f0-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:46:51.051258 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.051258    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b59894f-5aac-434f-a49e-5da3b179f6f0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:46:51.051416 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.051273    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khrdx\" (UniqueName: \"kubernetes.io/projected/0b59894f-5aac-434f-a49e-5da3b179f6f0-kube-api-access-khrdx\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:46:51.051416 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.051285    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b59894f-5aac-434f-a49e-5da3b179f6f0-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:46:51.051416 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.051297    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b59894f-5aac-434f-a49e-5da3b179f6f0-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:46:51.267121 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.267008    2562 generic.go:356] "Generic (PLEG): container finished" podID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerID="a5aff21265bf204040fe4ed18385692cf629a0f6a313003a36fed60b432be498" exitCode=0
Jul 21 18:46:51.267121 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.267085    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"
Jul 21 18:46:51.267121 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.267104    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" event={"ID":"0b59894f-5aac-434f-a49e-5da3b179f6f0","Type":"ContainerDied","Data":"a5aff21265bf204040fe4ed18385692cf629a0f6a313003a36fed60b432be498"}
Jul 21 18:46:51.267423 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.267179    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t" event={"ID":"0b59894f-5aac-434f-a49e-5da3b179f6f0","Type":"ContainerDied","Data":"6e99004fdbed4f30757e6d7e0281bfb928c0f4a3097f683318058762b9c17ef6"}
Jul 21 18:46:51.267423 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.267217    2562 scope.go:117] "RemoveContainer" containerID="902b72c6c517316489e43937189354a50184421898f1ea5da2b61b3183a7b9fa"
Jul 21 18:46:51.276038 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.276022    2562 scope.go:117] "RemoveContainer" containerID="a5aff21265bf204040fe4ed18385692cf629a0f6a313003a36fed60b432be498"
Jul 21 18:46:51.283606 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.283590    2562 scope.go:117] "RemoveContainer" containerID="c8bcd0cb2d39e260304d3b28f46da4d90d5e562e65e6d7cb5440d7fbbc2e6288"
Jul 21 18:46:51.288529 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.288511    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"]
Jul 21 18:46:51.291660 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.291643    2562 scope.go:117] "RemoveContainer" containerID="902b72c6c517316489e43937189354a50184421898f1ea5da2b61b3183a7b9fa"
Jul 21 18:46:51.291906 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:46:51.291888    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902b72c6c517316489e43937189354a50184421898f1ea5da2b61b3183a7b9fa\": container with ID starting with 902b72c6c517316489e43937189354a50184421898f1ea5da2b61b3183a7b9fa not found: ID does not exist" containerID="902b72c6c517316489e43937189354a50184421898f1ea5da2b61b3183a7b9fa"
Jul 21 18:46:51.291972 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.291925    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902b72c6c517316489e43937189354a50184421898f1ea5da2b61b3183a7b9fa"} err="failed to get container status \"902b72c6c517316489e43937189354a50184421898f1ea5da2b61b3183a7b9fa\": rpc error: code = NotFound desc = could not find container \"902b72c6c517316489e43937189354a50184421898f1ea5da2b61b3183a7b9fa\": container with ID starting with 902b72c6c517316489e43937189354a50184421898f1ea5da2b61b3183a7b9fa not found: ID does not exist"
Jul 21 18:46:51.291972 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.291943    2562 scope.go:117] "RemoveContainer" containerID="a5aff21265bf204040fe4ed18385692cf629a0f6a313003a36fed60b432be498"
Jul 21 18:46:51.292227 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:46:51.292208    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5aff21265bf204040fe4ed18385692cf629a0f6a313003a36fed60b432be498\": container with ID starting with a5aff21265bf204040fe4ed18385692cf629a0f6a313003a36fed60b432be498 not found: ID does not exist" containerID="a5aff21265bf204040fe4ed18385692cf629a0f6a313003a36fed60b432be498"
Jul 21 18:46:51.292298 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.292232    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5aff21265bf204040fe4ed18385692cf629a0f6a313003a36fed60b432be498"} err="failed to get container status \"a5aff21265bf204040fe4ed18385692cf629a0f6a313003a36fed60b432be498\": rpc error: code = NotFound desc = could not find container \"a5aff21265bf204040fe4ed18385692cf629a0f6a313003a36fed60b432be498\": container with ID starting with a5aff21265bf204040fe4ed18385692cf629a0f6a313003a36fed60b432be498 not found: ID does not exist"
Jul 21 18:46:51.292298 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.292247    2562 scope.go:117] "RemoveContainer" containerID="c8bcd0cb2d39e260304d3b28f46da4d90d5e562e65e6d7cb5440d7fbbc2e6288"
Jul 21 18:46:51.292401 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.292307    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-84994546b5-fsb5t"]
Jul 21 18:46:51.292512 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:46:51.292494    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8bcd0cb2d39e260304d3b28f46da4d90d5e562e65e6d7cb5440d7fbbc2e6288\": container with ID starting with c8bcd0cb2d39e260304d3b28f46da4d90d5e562e65e6d7cb5440d7fbbc2e6288 not found: ID does not exist" containerID="c8bcd0cb2d39e260304d3b28f46da4d90d5e562e65e6d7cb5440d7fbbc2e6288"
Jul 21 18:46:51.292569 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:51.292520    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8bcd0cb2d39e260304d3b28f46da4d90d5e562e65e6d7cb5440d7fbbc2e6288"} err="failed to get container status \"c8bcd0cb2d39e260304d3b28f46da4d90d5e562e65e6d7cb5440d7fbbc2e6288\": rpc error: code = NotFound desc = could not find container \"c8bcd0cb2d39e260304d3b28f46da4d90d5e562e65e6d7cb5440d7fbbc2e6288\": container with ID starting with c8bcd0cb2d39e260304d3b28f46da4d90d5e562e65e6d7cb5440d7fbbc2e6288 not found: ID does not exist"
Jul 21 18:46:52.921225 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:52.921180    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" path="/var/lib/kubelet/pods/0b59894f-5aac-434f-a49e-5da3b179f6f0/volumes"
Jul 21 18:46:53.277953 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:53.277927    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b_c3348dec-ec30-4f66-b710-d8e19ea037e6/storage-initializer/0.log"
Jul 21 18:46:53.278139 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:53.277964    2562 generic.go:356] "Generic (PLEG): container finished" podID="c3348dec-ec30-4f66-b710-d8e19ea037e6" containerID="0496925978b5cec05a4ce231cb8c4814e1c1a7844ee6bf8282b69de51fff02f2" exitCode=1
Jul 21 18:46:53.278139 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:53.278027    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b" event={"ID":"c3348dec-ec30-4f66-b710-d8e19ea037e6","Type":"ContainerDied","Data":"0496925978b5cec05a4ce231cb8c4814e1c1a7844ee6bf8282b69de51fff02f2"}
Jul 21 18:46:54.283701 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:54.283671    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b_c3348dec-ec30-4f66-b710-d8e19ea037e6/storage-initializer/0.log"
Jul 21 18:46:54.284099 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:54.283774    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b" event={"ID":"c3348dec-ec30-4f66-b710-d8e19ea037e6","Type":"ContainerStarted","Data":"d329c9cf842936fafcf6a02df4af3dc6b58f801a4fa3b62fee568281495129ca"}
Jul 21 18:46:57.299351 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:57.299281    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b_c3348dec-ec30-4f66-b710-d8e19ea037e6/storage-initializer/1.log"
Jul 21 18:46:57.299742 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:57.299634    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b_c3348dec-ec30-4f66-b710-d8e19ea037e6/storage-initializer/0.log"
Jul 21 18:46:57.299742 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:57.299670    2562 generic.go:356] "Generic (PLEG): container finished" podID="c3348dec-ec30-4f66-b710-d8e19ea037e6" containerID="d329c9cf842936fafcf6a02df4af3dc6b58f801a4fa3b62fee568281495129ca" exitCode=1
Jul 21 18:46:57.299811 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:57.299744    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b" event={"ID":"c3348dec-ec30-4f66-b710-d8e19ea037e6","Type":"ContainerDied","Data":"d329c9cf842936fafcf6a02df4af3dc6b58f801a4fa3b62fee568281495129ca"}
Jul 21 18:46:57.299843 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:57.299813    2562 scope.go:117] "RemoveContainer" containerID="0496925978b5cec05a4ce231cb8c4814e1c1a7844ee6bf8282b69de51fff02f2"
Jul 21 18:46:57.300168 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:57.300146    2562 scope.go:117] "RemoveContainer" containerID="0496925978b5cec05a4ce231cb8c4814e1c1a7844ee6bf8282b69de51fff02f2"
Jul 21 18:46:57.311035 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:46:57.311009    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b_kserve-ci-e2e-test(c3348dec-ec30-4f66-b710-d8e19ea037e6)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b" podUID="c3348dec-ec30-4f66-b710-d8e19ea037e6"
Jul 21 18:46:57.311167 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:46:57.311139    2562 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b_kserve-ci-e2e-test_c3348dec-ec30-4f66-b710-d8e19ea037e6_0 in pod sandbox ee31cfc3259a24c3fdd88727a2412131c6b83a47166e6d1f94ce80a76624a012 from index: no such id: '0496925978b5cec05a4ce231cb8c4814e1c1a7844ee6bf8282b69de51fff02f2'" containerID="0496925978b5cec05a4ce231cb8c4814e1c1a7844ee6bf8282b69de51fff02f2"
Jul 21 18:46:57.311218 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:57.311177    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0496925978b5cec05a4ce231cb8c4814e1c1a7844ee6bf8282b69de51fff02f2"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b_kserve-ci-e2e-test_c3348dec-ec30-4f66-b710-d8e19ea037e6_0 in pod sandbox ee31cfc3259a24c3fdd88727a2412131c6b83a47166e6d1f94ce80a76624a012 from index: no such id: '0496925978b5cec05a4ce231cb8c4814e1c1a7844ee6bf8282b69de51fff02f2'"
Jul 21 18:46:57.352918 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:57.352888    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"]
Jul 21 18:46:57.395725 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:46:57.395696    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 21 18:46:57.395868 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:46:57.395779    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3348dec-ec30-4f66-b710-d8e19ea037e6-proxy-tls podName:c3348dec-ec30-4f66-b710-d8e19ea037e6 nodeName:}" failed. No retries permitted until 2026-07-21 18:46:57.895757446 +0000 UTC m=+4869.467379183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c3348dec-ec30-4f66-b710-d8e19ea037e6-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b" (UID: "c3348dec-ec30-4f66-b710-d8e19ea037e6") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 21 18:46:57.898966 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:46:57.898935    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 21 18:46:57.899154 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:46:57.899009    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3348dec-ec30-4f66-b710-d8e19ea037e6-proxy-tls podName:c3348dec-ec30-4f66-b710-d8e19ea037e6 nodeName:}" failed. No retries permitted until 2026-07-21 18:46:58.898994764 +0000 UTC m=+4870.470616497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c3348dec-ec30-4f66-b710-d8e19ea037e6-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b" (UID: "c3348dec-ec30-4f66-b710-d8e19ea037e6") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 21 18:46:58.306033 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.306007    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b_c3348dec-ec30-4f66-b710-d8e19ea037e6/storage-initializer/1.log"
Jul 21 18:46:58.440330 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.440306    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b_c3348dec-ec30-4f66-b710-d8e19ea037e6/storage-initializer/1.log"
Jul 21 18:46:58.443788 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.443766    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"
Jul 21 18:46:58.466328 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.466300    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"]
Jul 21 18:46:58.466619 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.466605    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="storage-initializer"
Jul 21 18:46:58.466619 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.466620    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="storage-initializer"
Jul 21 18:46:58.466718 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.466640    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kserve-container"
Jul 21 18:46:58.466718 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.466646    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kserve-container"
Jul 21 18:46:58.466718 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.466656    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3348dec-ec30-4f66-b710-d8e19ea037e6" containerName="storage-initializer"
Jul 21 18:46:58.466718 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.466661    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3348dec-ec30-4f66-b710-d8e19ea037e6" containerName="storage-initializer"
Jul 21 18:46:58.466718 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.466667    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3348dec-ec30-4f66-b710-d8e19ea037e6" containerName="storage-initializer"
Jul 21 18:46:58.466718 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.466672    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3348dec-ec30-4f66-b710-d8e19ea037e6" containerName="storage-initializer"
Jul 21 18:46:58.466718 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.466696    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kube-rbac-proxy"
Jul 21 18:46:58.466718 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.466702    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kube-rbac-proxy"
Jul 21 18:46:58.466998 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.466746    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kserve-container"
Jul 21 18:46:58.466998 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.466752    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3348dec-ec30-4f66-b710-d8e19ea037e6" containerName="storage-initializer"
Jul 21 18:46:58.466998 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.466760    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b59894f-5aac-434f-a49e-5da3b179f6f0" containerName="kube-rbac-proxy"
Jul 21 18:46:58.466998 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.466853    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3348dec-ec30-4f66-b710-d8e19ea037e6" containerName="storage-initializer"
Jul 21 18:46:58.469705 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.469691    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:58.472162 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.472140    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jul 21 18:46:58.472268 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.472179    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jul 21 18:46:58.472547 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.472527    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:46:58.480700 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.480680    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"]
Jul 21 18:46:58.503224 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.503196    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3348dec-ec30-4f66-b710-d8e19ea037e6-kserve-provision-location\") pod \"c3348dec-ec30-4f66-b710-d8e19ea037e6\" (UID: \"c3348dec-ec30-4f66-b710-d8e19ea037e6\") "
Jul 21 18:46:58.503348 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.503226    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8pnk\" (UniqueName: \"kubernetes.io/projected/c3348dec-ec30-4f66-b710-d8e19ea037e6-kube-api-access-r8pnk\") pod \"c3348dec-ec30-4f66-b710-d8e19ea037e6\" (UID: \"c3348dec-ec30-4f66-b710-d8e19ea037e6\") "
Jul 21 18:46:58.503348 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.503294    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3348dec-ec30-4f66-b710-d8e19ea037e6-kube-rbac-proxy-sar-config\") pod \"c3348dec-ec30-4f66-b710-d8e19ea037e6\" (UID: \"c3348dec-ec30-4f66-b710-d8e19ea037e6\") "
Jul 21 18:46:58.503348 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.503319    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3348dec-ec30-4f66-b710-d8e19ea037e6-proxy-tls\") pod \"c3348dec-ec30-4f66-b710-d8e19ea037e6\" (UID: \"c3348dec-ec30-4f66-b710-d8e19ea037e6\") "
Jul 21 18:46:58.503523 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.503411    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqxh6\" (UniqueName: \"kubernetes.io/projected/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kube-api-access-mqxh6\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:58.503523 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.503472    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3348dec-ec30-4f66-b710-d8e19ea037e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c3348dec-ec30-4f66-b710-d8e19ea037e6" (UID: "c3348dec-ec30-4f66-b710-d8e19ea037e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:46:58.503523 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.503496    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:58.503659 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.503532    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:58.503659 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.503601    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:58.503659 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.503637    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:58.503825 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.503665    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3348dec-ec30-4f66-b710-d8e19ea037e6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c3348dec-ec30-4f66-b710-d8e19ea037e6" (UID: "c3348dec-ec30-4f66-b710-d8e19ea037e6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:46:58.503825 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.503706    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3348dec-ec30-4f66-b710-d8e19ea037e6-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:46:58.505502 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.505465    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3348dec-ec30-4f66-b710-d8e19ea037e6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c3348dec-ec30-4f66-b710-d8e19ea037e6" (UID: "c3348dec-ec30-4f66-b710-d8e19ea037e6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:46:58.505643 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.505622    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3348dec-ec30-4f66-b710-d8e19ea037e6-kube-api-access-r8pnk" (OuterVolumeSpecName: "kube-api-access-r8pnk") pod "c3348dec-ec30-4f66-b710-d8e19ea037e6" (UID: "c3348dec-ec30-4f66-b710-d8e19ea037e6"). InnerVolumeSpecName "kube-api-access-r8pnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:46:58.604906 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.604835    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:58.604906 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.604871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:58.605094 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:46:58.604971    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 21 18:46:58.605094 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.604991    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:58.605094 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:46:58.605035    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-proxy-tls podName:5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d nodeName:}" failed. No retries permitted until 2026-07-21 18:46:59.10501438 +0000 UTC m=+4870.676636130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" (UID: "5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 21 18:46:58.605256 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.605093    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:58.605256 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.605156    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqxh6\" (UniqueName: \"kubernetes.io/projected/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kube-api-access-mqxh6\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:58.605256 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.605217    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3348dec-ec30-4f66-b710-d8e19ea037e6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:46:58.605256 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.605234    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3348dec-ec30-4f66-b710-d8e19ea037e6-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:46:58.605256 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.605248    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8pnk\" (UniqueName: \"kubernetes.io/projected/c3348dec-ec30-4f66-b710-d8e19ea037e6-kube-api-access-r8pnk\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:46:58.605444 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.605257    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:58.605588 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.605570    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:58.605624 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.605601    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:58.613540 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:58.613517    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqxh6\" (UniqueName: \"kubernetes.io/projected/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kube-api-access-mqxh6\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:59.109197 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:59.109162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:59.111605 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:59.111575    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:59.312352 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:59.312276    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b_c3348dec-ec30-4f66-b710-d8e19ea037e6/storage-initializer/1.log"
Jul 21 18:46:59.312763 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:59.312367    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b" event={"ID":"c3348dec-ec30-4f66-b710-d8e19ea037e6","Type":"ContainerDied","Data":"ee31cfc3259a24c3fdd88727a2412131c6b83a47166e6d1f94ce80a76624a012"}
Jul 21 18:46:59.312763 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:59.312398    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"
Jul 21 18:46:59.312763 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:59.312480    2562 scope.go:117] "RemoveContainer" containerID="d329c9cf842936fafcf6a02df4af3dc6b58f801a4fa3b62fee568281495129ca"
Jul 21 18:46:59.346930 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:59.346905    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"]
Jul 21 18:46:59.352379 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:59.352352    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-588584577c-jkp6b"]
Jul 21 18:46:59.380721 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:59.380699    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:46:59.513389 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:46:59.513362    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"]
Jul 21 18:46:59.516121 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:46:59.516091    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff31a0d_5f7b_49c6_9bbf_3c26b9505d8d.slice/crio-b776a41c05f6723cce6e469e5a680cdc32b3683531a44e23e20181093b7a2981 WatchSource:0}: Error finding container b776a41c05f6723cce6e469e5a680cdc32b3683531a44e23e20181093b7a2981: Status 404 returned error can't find the container with id b776a41c05f6723cce6e469e5a680cdc32b3683531a44e23e20181093b7a2981
Jul 21 18:47:00.320036 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:00.319996    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" event={"ID":"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d","Type":"ContainerStarted","Data":"7aca9ca43cd396b302a3bb89cb96f6e99700e2c5739f10cf78b438228b7afaf9"}
Jul 21 18:47:00.320447 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:00.320078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" event={"ID":"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d","Type":"ContainerStarted","Data":"b776a41c05f6723cce6e469e5a680cdc32b3683531a44e23e20181093b7a2981"}
Jul 21 18:47:00.921665 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:00.921621    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3348dec-ec30-4f66-b710-d8e19ea037e6" path="/var/lib/kubelet/pods/c3348dec-ec30-4f66-b710-d8e19ea037e6/volumes"
Jul 21 18:47:01.326004 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:01.325912    2562 generic.go:356] "Generic (PLEG): container finished" podID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerID="7aca9ca43cd396b302a3bb89cb96f6e99700e2c5739f10cf78b438228b7afaf9" exitCode=0
Jul 21 18:47:01.326004 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:01.325993    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" event={"ID":"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d","Type":"ContainerDied","Data":"7aca9ca43cd396b302a3bb89cb96f6e99700e2c5739f10cf78b438228b7afaf9"}
Jul 21 18:47:02.333134 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:02.333094    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" event={"ID":"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d","Type":"ContainerStarted","Data":"1aa3be2546a63831249951e817af4b8b7379766ae29aa20b41df0591beb9951c"}
Jul 21 18:47:02.333616 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:02.333153    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" event={"ID":"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d","Type":"ContainerStarted","Data":"61f354171411cbd7f1ab3b2903aafcd307acadcb7af51de58a6cb38c66357f58"}
Jul 21 18:47:02.333616 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:02.333224    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:47:02.355004 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:02.354962    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" podStartSLOduration=3.426233174 podStartE2EDuration="4.354948945s" totalImagesPullingTime="0s" totalInitContainerRuntime="928.715771ms" isStatefulPod=false podCreationTimestamp="2026-07-21 18:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:47:02.353362006 +0000 UTC m=+4873.924983782" watchObservedRunningTime="2026-07-21 18:47:02.354948945 +0000 UTC m=+4873.926570775"
Jul 21 18:47:03.337954 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:03.337911    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:47:03.339364 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:03.339337    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 21 18:47:04.341884 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:04.341841    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 21 18:47:09.347943 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:09.347914    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:47:09.348608 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:09.348575    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 21 18:47:19.349145 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:19.349105    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 21 18:47:29.348965 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:29.348927    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 21 18:47:39.349041 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:39.349002    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 21 18:47:49.348693 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:49.348651    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 21 18:47:59.348525 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:47:59.348486    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 21 18:48:09.349392 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:09.349311    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 21 18:48:19.349263 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:19.349234    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:48:28.562797 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:28.562764    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"]
Jul 21 18:48:28.563317 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:28.563030    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kserve-container" containerID="cri-o://61f354171411cbd7f1ab3b2903aafcd307acadcb7af51de58a6cb38c66357f58" gracePeriod=30
Jul 21 18:48:28.563317 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:28.563085    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kube-rbac-proxy" containerID="cri-o://1aa3be2546a63831249951e817af4b8b7379766ae29aa20b41df0591beb9951c" gracePeriod=30
Jul 21 18:48:28.706045 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:28.706017    2562 generic.go:356] "Generic (PLEG): container finished" podID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerID="1aa3be2546a63831249951e817af4b8b7379766ae29aa20b41df0591beb9951c" exitCode=2
Jul 21 18:48:28.706182 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:28.706085    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" event={"ID":"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d","Type":"ContainerDied","Data":"1aa3be2546a63831249951e817af4b8b7379766ae29aa20b41df0591beb9951c"}
Jul 21 18:48:29.342465 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.342423    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Jul 21 18:48:29.348833 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.348808    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 21 18:48:29.662868 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.662836    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"]
Jul 21 18:48:29.666301 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.666282    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"
Jul 21 18:48:29.669139 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.669116    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jul 21 18:48:29.669249 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.669122    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jul 21 18:48:29.679671 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.679653    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"]
Jul 21 18:48:29.726342 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.726319    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a56decbf-e844-4357-9175-5b8c72247864-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb\" (UID: \"a56decbf-e844-4357-9175-5b8c72247864\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"
Jul 21 18:48:29.726466 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.726355    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a56decbf-e844-4357-9175-5b8c72247864-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb\" (UID: \"a56decbf-e844-4357-9175-5b8c72247864\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"
Jul 21 18:48:29.726466 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.726393    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a56decbf-e844-4357-9175-5b8c72247864-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb\" (UID: \"a56decbf-e844-4357-9175-5b8c72247864\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"
Jul 21 18:48:29.726539 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.726484    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89frh\" (UniqueName: \"kubernetes.io/projected/a56decbf-e844-4357-9175-5b8c72247864-kube-api-access-89frh\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb\" (UID: \"a56decbf-e844-4357-9175-5b8c72247864\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"
Jul 21 18:48:29.827107 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.827056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89frh\" (UniqueName: \"kubernetes.io/projected/a56decbf-e844-4357-9175-5b8c72247864-kube-api-access-89frh\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb\" (UID: \"a56decbf-e844-4357-9175-5b8c72247864\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"
Jul 21 18:48:29.827278 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.827126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a56decbf-e844-4357-9175-5b8c72247864-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb\" (UID: \"a56decbf-e844-4357-9175-5b8c72247864\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"
Jul 21 18:48:29.827278 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.827153    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a56decbf-e844-4357-9175-5b8c72247864-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb\" (UID: \"a56decbf-e844-4357-9175-5b8c72247864\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"
Jul 21 18:48:29.827278 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.827184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a56decbf-e844-4357-9175-5b8c72247864-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb\" (UID: \"a56decbf-e844-4357-9175-5b8c72247864\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"
Jul 21 18:48:29.827625 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.827600    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a56decbf-e844-4357-9175-5b8c72247864-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb\" (UID: \"a56decbf-e844-4357-9175-5b8c72247864\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"
Jul 21 18:48:29.827784 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.827766    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a56decbf-e844-4357-9175-5b8c72247864-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb\" (UID: \"a56decbf-e844-4357-9175-5b8c72247864\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"
Jul 21 18:48:29.829682 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.829651    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a56decbf-e844-4357-9175-5b8c72247864-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb\" (UID: \"a56decbf-e844-4357-9175-5b8c72247864\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"
Jul 21 18:48:29.835827 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.835807    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89frh\" (UniqueName: \"kubernetes.io/projected/a56decbf-e844-4357-9175-5b8c72247864-kube-api-access-89frh\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb\" (UID: \"a56decbf-e844-4357-9175-5b8c72247864\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"
Jul 21 18:48:29.976280 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:29.976207    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"
Jul 21 18:48:30.109582 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:30.109559    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"]
Jul 21 18:48:30.111224 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:48:30.111190    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56decbf_e844_4357_9175_5b8c72247864.slice/crio-3f1c6713b5530fb680e1a86aae00f1efeb9a1c49092f3a0aa66af3de046fa8ab WatchSource:0}: Error finding container 3f1c6713b5530fb680e1a86aae00f1efeb9a1c49092f3a0aa66af3de046fa8ab: Status 404 returned error can't find the container with id 3f1c6713b5530fb680e1a86aae00f1efeb9a1c49092f3a0aa66af3de046fa8ab
Jul 21 18:48:30.716518 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:30.716477    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb" event={"ID":"a56decbf-e844-4357-9175-5b8c72247864","Type":"ContainerStarted","Data":"ed25c1a11bf82e1728f3587492948f4a2330221f2a43351b9584159b990182b3"}
Jul 21 18:48:30.716994 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:30.716554    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb" event={"ID":"a56decbf-e844-4357-9175-5b8c72247864","Type":"ContainerStarted","Data":"3f1c6713b5530fb680e1a86aae00f1efeb9a1c49092f3a0aa66af3de046fa8ab"}
Jul 21 18:48:33.012269 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.012240    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:48:33.057086 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.057006    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqxh6\" (UniqueName: \"kubernetes.io/projected/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kube-api-access-mqxh6\") pod \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") "
Jul 21 18:48:33.057086 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.057054    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kserve-provision-location\") pod \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") "
Jul 21 18:48:33.057247 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.057124    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-proxy-tls\") pod \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") "
Jul 21 18:48:33.057247 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.057176    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kube-rbac-proxy-sar-config\") pod \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") "
Jul 21 18:48:33.057247 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.057214    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-cabundle-cert\") pod \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\" (UID: \"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d\") "
Jul 21 18:48:33.057410 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.057351    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" (UID: "5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:48:33.057526 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.057505    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:48:33.057586 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.057534    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" (UID: "5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:48:33.057586 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.057571    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" (UID: "5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:48:33.059206 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.059186    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kube-api-access-mqxh6" (OuterVolumeSpecName: "kube-api-access-mqxh6") pod "5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" (UID: "5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d"). InnerVolumeSpecName "kube-api-access-mqxh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:48:33.059351 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.059335    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" (UID: "5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:48:33.157940 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.157919    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqxh6\" (UniqueName: \"kubernetes.io/projected/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kube-api-access-mqxh6\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:48:33.157940 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.157941    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:48:33.158107 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.157950    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:48:33.158107 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.157959    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:48:33.731454 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.731415    2562 generic.go:356] "Generic (PLEG): container finished" podID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerID="61f354171411cbd7f1ab3b2903aafcd307acadcb7af51de58a6cb38c66357f58" exitCode=0
Jul 21 18:48:33.731637 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.731493    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"
Jul 21 18:48:33.731637 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.731506    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" event={"ID":"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d","Type":"ContainerDied","Data":"61f354171411cbd7f1ab3b2903aafcd307acadcb7af51de58a6cb38c66357f58"}
Jul 21 18:48:33.731637 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.731596    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk" event={"ID":"5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d","Type":"ContainerDied","Data":"b776a41c05f6723cce6e469e5a680cdc32b3683531a44e23e20181093b7a2981"}
Jul 21 18:48:33.731637 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.731624    2562 scope.go:117] "RemoveContainer" containerID="1aa3be2546a63831249951e817af4b8b7379766ae29aa20b41df0591beb9951c"
Jul 21 18:48:33.740566 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.740551    2562 scope.go:117] "RemoveContainer" containerID="61f354171411cbd7f1ab3b2903aafcd307acadcb7af51de58a6cb38c66357f58"
Jul 21 18:48:33.747740 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.747726    2562 scope.go:117] "RemoveContainer" containerID="7aca9ca43cd396b302a3bb89cb96f6e99700e2c5739f10cf78b438228b7afaf9"
Jul 21 18:48:33.753706 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.753684    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"]
Jul 21 18:48:33.755891 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.755872    2562 scope.go:117] "RemoveContainer" containerID="1aa3be2546a63831249951e817af4b8b7379766ae29aa20b41df0591beb9951c"
Jul 21 18:48:33.756185 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:48:33.756161    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa3be2546a63831249951e817af4b8b7379766ae29aa20b41df0591beb9951c\": container with ID starting with 1aa3be2546a63831249951e817af4b8b7379766ae29aa20b41df0591beb9951c not found: ID does not exist" containerID="1aa3be2546a63831249951e817af4b8b7379766ae29aa20b41df0591beb9951c"
Jul 21 18:48:33.756289 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.756190    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa3be2546a63831249951e817af4b8b7379766ae29aa20b41df0591beb9951c"} err="failed to get container status \"1aa3be2546a63831249951e817af4b8b7379766ae29aa20b41df0591beb9951c\": rpc error: code = NotFound desc = could not find container \"1aa3be2546a63831249951e817af4b8b7379766ae29aa20b41df0591beb9951c\": container with ID starting with 1aa3be2546a63831249951e817af4b8b7379766ae29aa20b41df0591beb9951c not found: ID does not exist"
Jul 21 18:48:33.756289 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.756207    2562 scope.go:117] "RemoveContainer" containerID="61f354171411cbd7f1ab3b2903aafcd307acadcb7af51de58a6cb38c66357f58"
Jul 21 18:48:33.756500 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:48:33.756486    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f354171411cbd7f1ab3b2903aafcd307acadcb7af51de58a6cb38c66357f58\": container with ID starting with 61f354171411cbd7f1ab3b2903aafcd307acadcb7af51de58a6cb38c66357f58 not found: ID does not exist" containerID="61f354171411cbd7f1ab3b2903aafcd307acadcb7af51de58a6cb38c66357f58"
Jul 21 18:48:33.756558 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.756506    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f354171411cbd7f1ab3b2903aafcd307acadcb7af51de58a6cb38c66357f58"} err="failed to get container status \"61f354171411cbd7f1ab3b2903aafcd307acadcb7af51de58a6cb38c66357f58\": rpc error: code = NotFound desc = could not find container \"61f354171411cbd7f1ab3b2903aafcd307acadcb7af51de58a6cb38c66357f58\": container with ID starting with 61f354171411cbd7f1ab3b2903aafcd307acadcb7af51de58a6cb38c66357f58 not found: ID does not exist"
Jul 21 18:48:33.756558 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.756520    2562 scope.go:117] "RemoveContainer" containerID="7aca9ca43cd396b302a3bb89cb96f6e99700e2c5739f10cf78b438228b7afaf9"
Jul 21 18:48:33.756770 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:48:33.756752    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aca9ca43cd396b302a3bb89cb96f6e99700e2c5739f10cf78b438228b7afaf9\": container with ID starting with 7aca9ca43cd396b302a3bb89cb96f6e99700e2c5739f10cf78b438228b7afaf9 not found: ID does not exist" containerID="7aca9ca43cd396b302a3bb89cb96f6e99700e2c5739f10cf78b438228b7afaf9"
Jul 21 18:48:33.756811 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.756777    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aca9ca43cd396b302a3bb89cb96f6e99700e2c5739f10cf78b438228b7afaf9"} err="failed to get container status \"7aca9ca43cd396b302a3bb89cb96f6e99700e2c5739f10cf78b438228b7afaf9\": rpc error: code = NotFound desc = could not find container \"7aca9ca43cd396b302a3bb89cb96f6e99700e2c5739f10cf78b438228b7afaf9\": container with ID starting with 7aca9ca43cd396b302a3bb89cb96f6e99700e2c5739f10cf78b438228b7afaf9 not found: ID does not exist"
Jul 21 18:48:33.757378 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:33.757359    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6f5f576c85-qv9wk"]
Jul 21 18:48:34.737349 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:34.737320    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb_a56decbf-e844-4357-9175-5b8c72247864/storage-initializer/0.log"
Jul 21 18:48:34.737797 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:34.737358    2562 generic.go:356] "Generic (PLEG): container finished" podID="a56decbf-e844-4357-9175-5b8c72247864" containerID="ed25c1a11bf82e1728f3587492948f4a2330221f2a43351b9584159b990182b3" exitCode=1
Jul 21 18:48:34.737797 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:34.737426    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb" event={"ID":"a56decbf-e844-4357-9175-5b8c72247864","Type":"ContainerDied","Data":"ed25c1a11bf82e1728f3587492948f4a2330221f2a43351b9584159b990182b3"}
Jul 21 18:48:34.923862 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:34.923831    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" path="/var/lib/kubelet/pods/5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d/volumes"
Jul 21 18:48:35.744351 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:35.744322    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb_a56decbf-e844-4357-9175-5b8c72247864/storage-initializer/0.log"
Jul 21 18:48:35.744771 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:35.744439    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb" event={"ID":"a56decbf-e844-4357-9175-5b8c72247864","Type":"ContainerStarted","Data":"49dec7c535d4e746a8014c83e57a8b80c8a539c57bac9337ed116bc233c02494"}
Jul 21 18:48:39.633740 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:39.633702    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"]
Jul 21 18:48:39.634253 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:39.633904    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb" podUID="a56decbf-e844-4357-9175-5b8c72247864" containerName="storage-initializer" containerID="cri-o://49dec7c535d4e746a8014c83e57a8b80c8a539c57bac9337ed116bc233c02494" gracePeriod=30
Jul 21 18:48:40.376378 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.376358    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb_a56decbf-e844-4357-9175-5b8c72247864/storage-initializer/1.log"
Jul 21 18:48:40.376755 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.376740    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb_a56decbf-e844-4357-9175-5b8c72247864/storage-initializer/0.log"
Jul 21 18:48:40.378990 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.378975    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"
Jul 21 18:48:40.516130 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.516042    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a56decbf-e844-4357-9175-5b8c72247864-kube-rbac-proxy-sar-config\") pod \"a56decbf-e844-4357-9175-5b8c72247864\" (UID: \"a56decbf-e844-4357-9175-5b8c72247864\") "
Jul 21 18:48:40.516264 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.516141    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89frh\" (UniqueName: \"kubernetes.io/projected/a56decbf-e844-4357-9175-5b8c72247864-kube-api-access-89frh\") pod \"a56decbf-e844-4357-9175-5b8c72247864\" (UID: \"a56decbf-e844-4357-9175-5b8c72247864\") "
Jul 21 18:48:40.516264 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.516207    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a56decbf-e844-4357-9175-5b8c72247864-kserve-provision-location\") pod \"a56decbf-e844-4357-9175-5b8c72247864\" (UID: \"a56decbf-e844-4357-9175-5b8c72247864\") "
Jul 21 18:48:40.516264 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.516234    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a56decbf-e844-4357-9175-5b8c72247864-proxy-tls\") pod \"a56decbf-e844-4357-9175-5b8c72247864\" (UID: \"a56decbf-e844-4357-9175-5b8c72247864\") "
Jul 21 18:48:40.516433 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.516371    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a56decbf-e844-4357-9175-5b8c72247864-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a56decbf-e844-4357-9175-5b8c72247864" (UID: "a56decbf-e844-4357-9175-5b8c72247864"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:48:40.516433 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.516416    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56decbf-e844-4357-9175-5b8c72247864-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a56decbf-e844-4357-9175-5b8c72247864" (UID: "a56decbf-e844-4357-9175-5b8c72247864"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:48:40.516522 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.516431    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a56decbf-e844-4357-9175-5b8c72247864-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:48:40.518378 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.518357    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a56decbf-e844-4357-9175-5b8c72247864-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a56decbf-e844-4357-9175-5b8c72247864" (UID: "a56decbf-e844-4357-9175-5b8c72247864"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:48:40.518523 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.518506    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a56decbf-e844-4357-9175-5b8c72247864-kube-api-access-89frh" (OuterVolumeSpecName: "kube-api-access-89frh") pod "a56decbf-e844-4357-9175-5b8c72247864" (UID: "a56decbf-e844-4357-9175-5b8c72247864"). InnerVolumeSpecName "kube-api-access-89frh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:48:40.617116 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.617086    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a56decbf-e844-4357-9175-5b8c72247864-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:48:40.617116 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.617112    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a56decbf-e844-4357-9175-5b8c72247864-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:48:40.617116 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.617122    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89frh\" (UniqueName: \"kubernetes.io/projected/a56decbf-e844-4357-9175-5b8c72247864-kube-api-access-89frh\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:48:40.726728 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.726694    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"]
Jul 21 18:48:40.727112 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.726995    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kserve-container"
Jul 21 18:48:40.727112 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.727005    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kserve-container"
Jul 21 18:48:40.727112 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.727024    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kube-rbac-proxy"
Jul 21 18:48:40.727112 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.727031    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kube-rbac-proxy"
Jul 21 18:48:40.727112 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.727038    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a56decbf-e844-4357-9175-5b8c72247864" containerName="storage-initializer"
Jul 21 18:48:40.727112 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.727045    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a56decbf-e844-4357-9175-5b8c72247864" containerName="storage-initializer"
Jul 21 18:48:40.727112 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.727054    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="storage-initializer"
Jul 21 18:48:40.727112 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.727059    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="storage-initializer"
Jul 21 18:48:40.727112 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.727083    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a56decbf-e844-4357-9175-5b8c72247864" containerName="storage-initializer"
Jul 21 18:48:40.727112 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.727088    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a56decbf-e844-4357-9175-5b8c72247864" containerName="storage-initializer"
Jul 21 18:48:40.727446 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.727134    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a56decbf-e844-4357-9175-5b8c72247864" containerName="storage-initializer"
Jul 21 18:48:40.727446 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.727141    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kube-rbac-proxy"
Jul 21 18:48:40.727446 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.727148    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff31a0d-5f7b-49c6-9bbf-3c26b9505d8d" containerName="kserve-container"
Jul 21 18:48:40.727446 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.727153    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a56decbf-e844-4357-9175-5b8c72247864" containerName="storage-initializer"
Jul 21 18:48:40.736681 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.736654    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:40.739801 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.739776    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:48:40.739956 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.739779    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jul 21 18:48:40.739956 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.739827    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jul 21 18:48:40.740904 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.740883    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"]
Jul 21 18:48:40.767208 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.767143    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb_a56decbf-e844-4357-9175-5b8c72247864/storage-initializer/1.log"
Jul 21 18:48:40.767536 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.767521    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb_a56decbf-e844-4357-9175-5b8c72247864/storage-initializer/0.log"
Jul 21 18:48:40.767598 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.767553    2562 generic.go:356] "Generic (PLEG): container finished" podID="a56decbf-e844-4357-9175-5b8c72247864" containerID="49dec7c535d4e746a8014c83e57a8b80c8a539c57bac9337ed116bc233c02494" exitCode=1
Jul 21 18:48:40.767637 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.767606    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"
Jul 21 18:48:40.767637 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.767626    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb" event={"ID":"a56decbf-e844-4357-9175-5b8c72247864","Type":"ContainerDied","Data":"49dec7c535d4e746a8014c83e57a8b80c8a539c57bac9337ed116bc233c02494"}
Jul 21 18:48:40.767707 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.767674    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb" event={"ID":"a56decbf-e844-4357-9175-5b8c72247864","Type":"ContainerDied","Data":"3f1c6713b5530fb680e1a86aae00f1efeb9a1c49092f3a0aa66af3de046fa8ab"}
Jul 21 18:48:40.767707 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.767704    2562 scope.go:117] "RemoveContainer" containerID="49dec7c535d4e746a8014c83e57a8b80c8a539c57bac9337ed116bc233c02494"
Jul 21 18:48:40.776460 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.776435    2562 scope.go:117] "RemoveContainer" containerID="ed25c1a11bf82e1728f3587492948f4a2330221f2a43351b9584159b990182b3"
Jul 21 18:48:40.783876 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.783860    2562 scope.go:117] "RemoveContainer" containerID="49dec7c535d4e746a8014c83e57a8b80c8a539c57bac9337ed116bc233c02494"
Jul 21 18:48:40.784511 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:48:40.784490    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49dec7c535d4e746a8014c83e57a8b80c8a539c57bac9337ed116bc233c02494\": container with ID starting with 49dec7c535d4e746a8014c83e57a8b80c8a539c57bac9337ed116bc233c02494 not found: ID does not exist" containerID="49dec7c535d4e746a8014c83e57a8b80c8a539c57bac9337ed116bc233c02494"
Jul 21 18:48:40.784571 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.784519    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49dec7c535d4e746a8014c83e57a8b80c8a539c57bac9337ed116bc233c02494"} err="failed to get container status \"49dec7c535d4e746a8014c83e57a8b80c8a539c57bac9337ed116bc233c02494\": rpc error: code = NotFound desc = could not find container \"49dec7c535d4e746a8014c83e57a8b80c8a539c57bac9337ed116bc233c02494\": container with ID starting with 49dec7c535d4e746a8014c83e57a8b80c8a539c57bac9337ed116bc233c02494 not found: ID does not exist"
Jul 21 18:48:40.784571 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.784536    2562 scope.go:117] "RemoveContainer" containerID="ed25c1a11bf82e1728f3587492948f4a2330221f2a43351b9584159b990182b3"
Jul 21 18:48:40.784788 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:48:40.784772    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed25c1a11bf82e1728f3587492948f4a2330221f2a43351b9584159b990182b3\": container with ID starting with ed25c1a11bf82e1728f3587492948f4a2330221f2a43351b9584159b990182b3 not found: ID does not exist" containerID="ed25c1a11bf82e1728f3587492948f4a2330221f2a43351b9584159b990182b3"
Jul 21 18:48:40.784834 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.784793    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed25c1a11bf82e1728f3587492948f4a2330221f2a43351b9584159b990182b3"} err="failed to get container status \"ed25c1a11bf82e1728f3587492948f4a2330221f2a43351b9584159b990182b3\": rpc error: code = NotFound desc = could not find container \"ed25c1a11bf82e1728f3587492948f4a2330221f2a43351b9584159b990182b3\": container with ID starting with ed25c1a11bf82e1728f3587492948f4a2330221f2a43351b9584159b990182b3 not found: ID does not exist"
Jul 21 18:48:40.801958 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.801933    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"]
Jul 21 18:48:40.803558 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.803535    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7d87c6f8f-w9hxb"]
Jul 21 18:48:40.818110 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.818087    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f651d6af-74ac-4c16-af46-5c8d86a36233-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:40.818196 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.818146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f651d6af-74ac-4c16-af46-5c8d86a36233-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:40.818196 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.818187    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2kmc\" (UniqueName: \"kubernetes.io/projected/f651d6af-74ac-4c16-af46-5c8d86a36233-kube-api-access-w2kmc\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:40.818283 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.818206    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f651d6af-74ac-4c16-af46-5c8d86a36233-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:40.818283 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.818237    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f651d6af-74ac-4c16-af46-5c8d86a36233-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:40.918734 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.918706    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f651d6af-74ac-4c16-af46-5c8d86a36233-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:40.918734 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.918736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2kmc\" (UniqueName: \"kubernetes.io/projected/f651d6af-74ac-4c16-af46-5c8d86a36233-kube-api-access-w2kmc\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:40.918953 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.918757    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f651d6af-74ac-4c16-af46-5c8d86a36233-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:40.918953 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.918777    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f651d6af-74ac-4c16-af46-5c8d86a36233-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:40.918953 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.918814    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f651d6af-74ac-4c16-af46-5c8d86a36233-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:40.919311 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.919273    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f651d6af-74ac-4c16-af46-5c8d86a36233-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:40.919501 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.919484    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f651d6af-74ac-4c16-af46-5c8d86a36233-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:40.919620 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.919592    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f651d6af-74ac-4c16-af46-5c8d86a36233-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:40.921898 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.921878    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f651d6af-74ac-4c16-af46-5c8d86a36233-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:40.922029 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.922009    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a56decbf-e844-4357-9175-5b8c72247864" path="/var/lib/kubelet/pods/a56decbf-e844-4357-9175-5b8c72247864/volumes"
Jul 21 18:48:40.928016 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:40.927997    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2kmc\" (UniqueName: \"kubernetes.io/projected/f651d6af-74ac-4c16-af46-5c8d86a36233-kube-api-access-w2kmc\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:41.048172 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:41.048100    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:41.185626 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:41.185601    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"]
Jul 21 18:48:41.187578 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:48:41.187553    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf651d6af_74ac_4c16_af46_5c8d86a36233.slice/crio-589e183d8e3c5fb69a6099865f049bdd1de3c93d01859b84127f1fe8d9922f59 WatchSource:0}: Error finding container 589e183d8e3c5fb69a6099865f049bdd1de3c93d01859b84127f1fe8d9922f59: Status 404 returned error can't find the container with id 589e183d8e3c5fb69a6099865f049bdd1de3c93d01859b84127f1fe8d9922f59
Jul 21 18:48:41.775883 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:41.775851    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" event={"ID":"f651d6af-74ac-4c16-af46-5c8d86a36233","Type":"ContainerStarted","Data":"6da7a49a3ee0b122755ae51d21633593cd6e26dc209e6e174e3bc684b4ebfb6f"}
Jul 21 18:48:41.776301 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:41.775908    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" event={"ID":"f651d6af-74ac-4c16-af46-5c8d86a36233","Type":"ContainerStarted","Data":"589e183d8e3c5fb69a6099865f049bdd1de3c93d01859b84127f1fe8d9922f59"}
Jul 21 18:48:42.782330 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:42.782292    2562 generic.go:356] "Generic (PLEG): container finished" podID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerID="6da7a49a3ee0b122755ae51d21633593cd6e26dc209e6e174e3bc684b4ebfb6f" exitCode=0
Jul 21 18:48:42.784617 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:42.782345    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" event={"ID":"f651d6af-74ac-4c16-af46-5c8d86a36233","Type":"ContainerDied","Data":"6da7a49a3ee0b122755ae51d21633593cd6e26dc209e6e174e3bc684b4ebfb6f"}
Jul 21 18:48:43.789370 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:43.789338    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" event={"ID":"f651d6af-74ac-4c16-af46-5c8d86a36233","Type":"ContainerStarted","Data":"ecb50e3f827b58577d8258cbdafd6795d111f465f63fbb60ce4d095409e4ec23"}
Jul 21 18:48:43.789756 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:43.789391    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" event={"ID":"f651d6af-74ac-4c16-af46-5c8d86a36233","Type":"ContainerStarted","Data":"fe1928fa7d8e57055c21cde15e6d56c88c612b36cf6191216143a878375f0c5e"}
Jul 21 18:48:43.789756 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:43.789521    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:43.815732 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:43.812892    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" podStartSLOduration=2.905421386 podStartE2EDuration="3.81287336s" totalImagesPullingTime="0s" totalInitContainerRuntime="907.451974ms" isStatefulPod=false podCreationTimestamp="2026-07-21 18:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:48:43.808242841 +0000 UTC m=+4975.379864609" watchObservedRunningTime="2026-07-21 18:48:43.81287336 +0000 UTC m=+4975.384495116"
Jul 21 18:48:44.794241 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:44.794202    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:44.795906 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:44.795876    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 21 18:48:45.797883 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:45.797835    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 21 18:48:50.802937 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:50.802911    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:48:50.803398 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:48:50.803372    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 21 18:49:00.803623 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:49:00.803584    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 21 18:49:10.804310 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:49:10.804270    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 21 18:49:20.804141 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:49:20.804099    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 21 18:49:30.803984 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:49:30.803943    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 21 18:49:40.803526 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:49:40.803433    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 21 18:49:50.803540 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:49:50.803502    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 21 18:50:00.804232 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:00.804193    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:50:10.808454 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:10.808424    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"]
Jul 21 18:50:10.809390 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:10.808686    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kserve-container" containerID="cri-o://fe1928fa7d8e57055c21cde15e6d56c88c612b36cf6191216143a878375f0c5e" gracePeriod=30
Jul 21 18:50:10.809390 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:10.808747    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kube-rbac-proxy" containerID="cri-o://ecb50e3f827b58577d8258cbdafd6795d111f465f63fbb60ce4d095409e4ec23" gracePeriod=30
Jul 21 18:50:11.146735 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:11.146655    2562 generic.go:356] "Generic (PLEG): container finished" podID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerID="ecb50e3f827b58577d8258cbdafd6795d111f465f63fbb60ce4d095409e4ec23" exitCode=2
Jul 21 18:50:11.146735 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:11.146718    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" event={"ID":"f651d6af-74ac-4c16-af46-5c8d86a36233","Type":"ContainerDied","Data":"ecb50e3f827b58577d8258cbdafd6795d111f465f63fbb60ce4d095409e4ec23"}
Jul 21 18:50:11.892114 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:11.892078    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"]
Jul 21 18:50:11.895626 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:11.895606    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:11.898407 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:11.898387    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jul 21 18:50:11.898524 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:11.898418    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jul 21 18:50:11.907688 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:11.907664    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"]
Jul 21 18:50:12.008014 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:12.007972    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e2d0184-e683-47ab-b74e-b350662b0eba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:12.008234 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:12.008026    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdp64\" (UniqueName: \"kubernetes.io/projected/5e2d0184-e683-47ab-b74e-b350662b0eba-kube-api-access-kdp64\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:12.008234 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:12.008139    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e2d0184-e683-47ab-b74e-b350662b0eba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:12.008234 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:12.008183    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e2d0184-e683-47ab-b74e-b350662b0eba-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:12.108756 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:12.108721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e2d0184-e683-47ab-b74e-b350662b0eba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:12.108934 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:12.108766    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e2d0184-e683-47ab-b74e-b350662b0eba-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:12.108934 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:12.108797    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e2d0184-e683-47ab-b74e-b350662b0eba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:12.108934 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:12.108819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdp64\" (UniqueName: \"kubernetes.io/projected/5e2d0184-e683-47ab-b74e-b350662b0eba-kube-api-access-kdp64\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:12.108934 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:50:12.108889    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 21 18:50:12.109200 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:50:12.108975    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e2d0184-e683-47ab-b74e-b350662b0eba-proxy-tls podName:5e2d0184-e683-47ab-b74e-b350662b0eba nodeName:}" failed. No retries permitted until 2026-07-21 18:50:12.608954128 +0000 UTC m=+5064.180575861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5e2d0184-e683-47ab-b74e-b350662b0eba-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf" (UID: "5e2d0184-e683-47ab-b74e-b350662b0eba") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 21 18:50:12.109272 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:12.109258    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e2d0184-e683-47ab-b74e-b350662b0eba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:12.109517 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:12.109495    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e2d0184-e683-47ab-b74e-b350662b0eba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:12.119682 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:12.119660    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdp64\" (UniqueName: \"kubernetes.io/projected/5e2d0184-e683-47ab-b74e-b350662b0eba-kube-api-access-kdp64\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:12.613774 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:12.613735    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e2d0184-e683-47ab-b74e-b350662b0eba-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:12.616260 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:12.616242    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e2d0184-e683-47ab-b74e-b350662b0eba-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:12.806198 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:12.806152    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:12.936634 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:12.936571    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"]
Jul 21 18:50:12.938850 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:50:12.938817    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2d0184_e683_47ab_b74e_b350662b0eba.slice/crio-620d33df81daad86fb7cb784289733370f5c0f7417c7662da359f09ed5aade0b WatchSource:0}: Error finding container 620d33df81daad86fb7cb784289733370f5c0f7417c7662da359f09ed5aade0b: Status 404 returned error can't find the container with id 620d33df81daad86fb7cb784289733370f5c0f7417c7662da359f09ed5aade0b
Jul 21 18:50:13.156850 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:13.156816    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf" event={"ID":"5e2d0184-e683-47ab-b74e-b350662b0eba","Type":"ContainerStarted","Data":"c9423c533314e21051dd7495bdecb6b5fd5386641c8c912e17b6f9b8fba683ce"}
Jul 21 18:50:13.157015 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:13.156869    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf" event={"ID":"5e2d0184-e683-47ab-b74e-b350662b0eba","Type":"ContainerStarted","Data":"620d33df81daad86fb7cb784289733370f5c0f7417c7662da359f09ed5aade0b"}
Jul 21 18:50:15.168398 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.168365    2562 generic.go:356] "Generic (PLEG): container finished" podID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerID="fe1928fa7d8e57055c21cde15e6d56c88c612b36cf6191216143a878375f0c5e" exitCode=0
Jul 21 18:50:15.168707 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.168421    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" event={"ID":"f651d6af-74ac-4c16-af46-5c8d86a36233","Type":"ContainerDied","Data":"fe1928fa7d8e57055c21cde15e6d56c88c612b36cf6191216143a878375f0c5e"}
Jul 21 18:50:15.168707 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.168465    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq" event={"ID":"f651d6af-74ac-4c16-af46-5c8d86a36233","Type":"ContainerDied","Data":"589e183d8e3c5fb69a6099865f049bdd1de3c93d01859b84127f1fe8d9922f59"}
Jul 21 18:50:15.168707 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.168485    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="589e183d8e3c5fb69a6099865f049bdd1de3c93d01859b84127f1fe8d9922f59"
Jul 21 18:50:15.169390 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.169373    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:50:15.338115 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.338018    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2kmc\" (UniqueName: \"kubernetes.io/projected/f651d6af-74ac-4c16-af46-5c8d86a36233-kube-api-access-w2kmc\") pod \"f651d6af-74ac-4c16-af46-5c8d86a36233\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") "
Jul 21 18:50:15.338115 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.338052    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f651d6af-74ac-4c16-af46-5c8d86a36233-cabundle-cert\") pod \"f651d6af-74ac-4c16-af46-5c8d86a36233\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") "
Jul 21 18:50:15.338115 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.338098    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f651d6af-74ac-4c16-af46-5c8d86a36233-kube-rbac-proxy-sar-config\") pod \"f651d6af-74ac-4c16-af46-5c8d86a36233\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") "
Jul 21 18:50:15.338409 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.338137    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f651d6af-74ac-4c16-af46-5c8d86a36233-kserve-provision-location\") pod \"f651d6af-74ac-4c16-af46-5c8d86a36233\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") "
Jul 21 18:50:15.338409 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.338180    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f651d6af-74ac-4c16-af46-5c8d86a36233-proxy-tls\") pod \"f651d6af-74ac-4c16-af46-5c8d86a36233\" (UID: \"f651d6af-74ac-4c16-af46-5c8d86a36233\") "
Jul 21 18:50:15.338521 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.338467    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f651d6af-74ac-4c16-af46-5c8d86a36233-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f651d6af-74ac-4c16-af46-5c8d86a36233" (UID: "f651d6af-74ac-4c16-af46-5c8d86a36233"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:50:15.338521 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.338507    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f651d6af-74ac-4c16-af46-5c8d86a36233-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f651d6af-74ac-4c16-af46-5c8d86a36233" (UID: "f651d6af-74ac-4c16-af46-5c8d86a36233"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:50:15.338602 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.338528    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f651d6af-74ac-4c16-af46-5c8d86a36233-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f651d6af-74ac-4c16-af46-5c8d86a36233" (UID: "f651d6af-74ac-4c16-af46-5c8d86a36233"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:50:15.340320 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.340300    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f651d6af-74ac-4c16-af46-5c8d86a36233-kube-api-access-w2kmc" (OuterVolumeSpecName: "kube-api-access-w2kmc") pod "f651d6af-74ac-4c16-af46-5c8d86a36233" (UID: "f651d6af-74ac-4c16-af46-5c8d86a36233"). InnerVolumeSpecName "kube-api-access-w2kmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:50:15.340381 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.340322    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f651d6af-74ac-4c16-af46-5c8d86a36233-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f651d6af-74ac-4c16-af46-5c8d86a36233" (UID: "f651d6af-74ac-4c16-af46-5c8d86a36233"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:50:15.438895 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.438865    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2kmc\" (UniqueName: \"kubernetes.io/projected/f651d6af-74ac-4c16-af46-5c8d86a36233-kube-api-access-w2kmc\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:50:15.438895 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.438889    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f651d6af-74ac-4c16-af46-5c8d86a36233-cabundle-cert\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:50:15.438895 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.438900    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f651d6af-74ac-4c16-af46-5c8d86a36233-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:50:15.439130 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.438908    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f651d6af-74ac-4c16-af46-5c8d86a36233-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:50:15.439130 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:15.438917    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f651d6af-74ac-4c16-af46-5c8d86a36233-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:50:16.172849 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:16.172817    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"
Jul 21 18:50:16.193663 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:16.193638    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"]
Jul 21 18:50:16.197725 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:16.197703    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7789c94898-fxkrq"]
Jul 21 18:50:16.921452 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:16.921418    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" path="/var/lib/kubelet/pods/f651d6af-74ac-4c16-af46-5c8d86a36233/volumes"
Jul 21 18:50:19.187537 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:19.187509    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf_5e2d0184-e683-47ab-b74e-b350662b0eba/storage-initializer/0.log"
Jul 21 18:50:19.187921 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:19.187549    2562 generic.go:356] "Generic (PLEG): container finished" podID="5e2d0184-e683-47ab-b74e-b350662b0eba" containerID="c9423c533314e21051dd7495bdecb6b5fd5386641c8c912e17b6f9b8fba683ce" exitCode=1
Jul 21 18:50:19.187921 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:19.187625    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf" event={"ID":"5e2d0184-e683-47ab-b74e-b350662b0eba","Type":"ContainerDied","Data":"c9423c533314e21051dd7495bdecb6b5fd5386641c8c912e17b6f9b8fba683ce"}
Jul 21 18:50:20.193934 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:20.193909    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf_5e2d0184-e683-47ab-b74e-b350662b0eba/storage-initializer/0.log"
Jul 21 18:50:20.194342 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:20.193993    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf" event={"ID":"5e2d0184-e683-47ab-b74e-b350662b0eba","Type":"ContainerStarted","Data":"ad43900c641c5d87c70bc5d44c66cd26cbfa694955b6e0d257b7f2839a7dc7c7"}
Jul 21 18:50:21.875118 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:21.875059    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"]
Jul 21 18:50:21.875547 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:21.875287    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf" podUID="5e2d0184-e683-47ab-b74e-b350662b0eba" containerName="storage-initializer" containerID="cri-o://ad43900c641c5d87c70bc5d44c66cd26cbfa694955b6e0d257b7f2839a7dc7c7" gracePeriod=30
Jul 21 18:50:23.427567 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:23.427542    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf_5e2d0184-e683-47ab-b74e-b350662b0eba/storage-initializer/1.log"
Jul 21 18:50:23.427960 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:23.427942    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf_5e2d0184-e683-47ab-b74e-b350662b0eba/storage-initializer/0.log"
Jul 21 18:50:23.430318 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:23.430299    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:23.608138 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:23.608033    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdp64\" (UniqueName: \"kubernetes.io/projected/5e2d0184-e683-47ab-b74e-b350662b0eba-kube-api-access-kdp64\") pod \"5e2d0184-e683-47ab-b74e-b350662b0eba\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") "
Jul 21 18:50:23.608138 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:23.608105    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e2d0184-e683-47ab-b74e-b350662b0eba-kube-rbac-proxy-sar-config\") pod \"5e2d0184-e683-47ab-b74e-b350662b0eba\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") "
Jul 21 18:50:23.608344 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:23.608151    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e2d0184-e683-47ab-b74e-b350662b0eba-kserve-provision-location\") pod \"5e2d0184-e683-47ab-b74e-b350662b0eba\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") "
Jul 21 18:50:23.608344 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:23.608185    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e2d0184-e683-47ab-b74e-b350662b0eba-proxy-tls\") pod \"5e2d0184-e683-47ab-b74e-b350662b0eba\" (UID: \"5e2d0184-e683-47ab-b74e-b350662b0eba\") "
Jul 21 18:50:23.608440 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:23.608361    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2d0184-e683-47ab-b74e-b350662b0eba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e2d0184-e683-47ab-b74e-b350662b0eba" (UID: "5e2d0184-e683-47ab-b74e-b350662b0eba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:50:23.608479 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:23.608436    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e2d0184-e683-47ab-b74e-b350662b0eba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5e2d0184-e683-47ab-b74e-b350662b0eba" (UID: "5e2d0184-e683-47ab-b74e-b350662b0eba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:50:23.610328 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:23.610299    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e2d0184-e683-47ab-b74e-b350662b0eba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5e2d0184-e683-47ab-b74e-b350662b0eba" (UID: "5e2d0184-e683-47ab-b74e-b350662b0eba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:50:23.610449 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:23.610356    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2d0184-e683-47ab-b74e-b350662b0eba-kube-api-access-kdp64" (OuterVolumeSpecName: "kube-api-access-kdp64") pod "5e2d0184-e683-47ab-b74e-b350662b0eba" (UID: "5e2d0184-e683-47ab-b74e-b350662b0eba"). InnerVolumeSpecName "kube-api-access-kdp64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:50:23.709459 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:23.709436    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e2d0184-e683-47ab-b74e-b350662b0eba-kserve-provision-location\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:50:23.709459 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:23.709460    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e2d0184-e683-47ab-b74e-b350662b0eba-proxy-tls\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:50:23.709625 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:23.709471    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdp64\" (UniqueName: \"kubernetes.io/projected/5e2d0184-e683-47ab-b74e-b350662b0eba-kube-api-access-kdp64\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:50:23.709625 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:23.709479    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e2d0184-e683-47ab-b74e-b350662b0eba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-228.ec2.internal\" DevicePath \"\""
Jul 21 18:50:24.214015 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:24.213991    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf_5e2d0184-e683-47ab-b74e-b350662b0eba/storage-initializer/1.log"
Jul 21 18:50:24.214377 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:24.214363    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf_5e2d0184-e683-47ab-b74e-b350662b0eba/storage-initializer/0.log"
Jul 21 18:50:24.214452 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:24.214400    2562 generic.go:356] "Generic (PLEG): container finished" podID="5e2d0184-e683-47ab-b74e-b350662b0eba" containerID="ad43900c641c5d87c70bc5d44c66cd26cbfa694955b6e0d257b7f2839a7dc7c7" exitCode=1
Jul 21 18:50:24.214505 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:24.214464    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"
Jul 21 18:50:24.214563 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:24.214501    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf" event={"ID":"5e2d0184-e683-47ab-b74e-b350662b0eba","Type":"ContainerDied","Data":"ad43900c641c5d87c70bc5d44c66cd26cbfa694955b6e0d257b7f2839a7dc7c7"}
Jul 21 18:50:24.214619 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:24.214581    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf" event={"ID":"5e2d0184-e683-47ab-b74e-b350662b0eba","Type":"ContainerDied","Data":"620d33df81daad86fb7cb784289733370f5c0f7417c7662da359f09ed5aade0b"}
Jul 21 18:50:24.214672 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:24.214624    2562 scope.go:117] "RemoveContainer" containerID="ad43900c641c5d87c70bc5d44c66cd26cbfa694955b6e0d257b7f2839a7dc7c7"
Jul 21 18:50:24.223290 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:24.223275    2562 scope.go:117] "RemoveContainer" containerID="c9423c533314e21051dd7495bdecb6b5fd5386641c8c912e17b6f9b8fba683ce"
Jul 21 18:50:24.230423 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:24.230404    2562 scope.go:117] "RemoveContainer" containerID="ad43900c641c5d87c70bc5d44c66cd26cbfa694955b6e0d257b7f2839a7dc7c7"
Jul 21 18:50:24.230663 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:50:24.230646    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad43900c641c5d87c70bc5d44c66cd26cbfa694955b6e0d257b7f2839a7dc7c7\": container with ID starting with ad43900c641c5d87c70bc5d44c66cd26cbfa694955b6e0d257b7f2839a7dc7c7 not found: ID does not exist" containerID="ad43900c641c5d87c70bc5d44c66cd26cbfa694955b6e0d257b7f2839a7dc7c7"
Jul 21 18:50:24.230722 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:24.230670    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad43900c641c5d87c70bc5d44c66cd26cbfa694955b6e0d257b7f2839a7dc7c7"} err="failed to get container status \"ad43900c641c5d87c70bc5d44c66cd26cbfa694955b6e0d257b7f2839a7dc7c7\": rpc error: code = NotFound desc = could not find container \"ad43900c641c5d87c70bc5d44c66cd26cbfa694955b6e0d257b7f2839a7dc7c7\": container with ID starting with ad43900c641c5d87c70bc5d44c66cd26cbfa694955b6e0d257b7f2839a7dc7c7 not found: ID does not exist"
Jul 21 18:50:24.230722 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:24.230688    2562 scope.go:117] "RemoveContainer" containerID="c9423c533314e21051dd7495bdecb6b5fd5386641c8c912e17b6f9b8fba683ce"
Jul 21 18:50:24.230893 ip-10-0-135-228 kubenswrapper[2562]: E0721 18:50:24.230866    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9423c533314e21051dd7495bdecb6b5fd5386641c8c912e17b6f9b8fba683ce\": container with ID starting with c9423c533314e21051dd7495bdecb6b5fd5386641c8c912e17b6f9b8fba683ce not found: ID does not exist" containerID="c9423c533314e21051dd7495bdecb6b5fd5386641c8c912e17b6f9b8fba683ce"
Jul 21 18:50:24.230932 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:24.230897    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9423c533314e21051dd7495bdecb6b5fd5386641c8c912e17b6f9b8fba683ce"} err="failed to get container status \"c9423c533314e21051dd7495bdecb6b5fd5386641c8c912e17b6f9b8fba683ce\": rpc error: code = NotFound desc = could not find container \"c9423c533314e21051dd7495bdecb6b5fd5386641c8c912e17b6f9b8fba683ce\": container with ID starting with c9423c533314e21051dd7495bdecb6b5fd5386641c8c912e17b6f9b8fba683ce not found: ID does not exist"
Jul 21 18:50:24.250268 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:24.250241    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"]
Jul 21 18:50:24.253891 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:24.253868    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6969d5d7bd-dtzcf"]
Jul 21 18:50:24.921344 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:24.921301    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e2d0184-e683-47ab-b74e-b350662b0eba" path="/var/lib/kubelet/pods/5e2d0184-e683-47ab-b74e-b350662b0eba/volumes"
Jul 21 18:50:59.411032 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:59.411004    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qht4v_e5718cfc-649b-4cd1-bd58-0ad5494c5e56/global-pull-secret-syncer/0.log"
Jul 21 18:50:59.607132 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:59.607105    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zl482_a9467694-b4aa-4b01-8a10-05dceaebc511/konnectivity-agent/0.log"
Jul 21 18:50:59.658725 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:50:59.658695    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-228.ec2.internal_5c1883a5f27b82c72d9daa1f8b8a95db/haproxy/0.log"
Jul 21 18:51:03.796769 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:03.796740    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-sjjsv_b0b85072-48eb-4ac9-a1a4-90baa59d26c4/monitoring-plugin/0.log"
Jul 21 18:51:04.015831 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:04.015801    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zdbqk_27b86f8c-114a-4875-ac3f-29fa472ca8ef/node-exporter/0.log"
Jul 21 18:51:04.040920 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:04.040894    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zdbqk_27b86f8c-114a-4875-ac3f-29fa472ca8ef/kube-rbac-proxy/0.log"
Jul 21 18:51:04.063840 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:04.063783    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zdbqk_27b86f8c-114a-4875-ac3f-29fa472ca8ef/init-textfile/0.log"
Jul 21 18:51:04.608017 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:04.607966    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bc74c496-s4xs8_94326f12-be7e-45fc-a6c3-8e1709c96b02/thanos-query/0.log"
Jul 21 18:51:04.630061 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:04.630038    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bc74c496-s4xs8_94326f12-be7e-45fc-a6c3-8e1709c96b02/kube-rbac-proxy-web/0.log"
Jul 21 18:51:04.651689 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:04.651657    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bc74c496-s4xs8_94326f12-be7e-45fc-a6c3-8e1709c96b02/kube-rbac-proxy/0.log"
Jul 21 18:51:04.673696 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:04.673670    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bc74c496-s4xs8_94326f12-be7e-45fc-a6c3-8e1709c96b02/prom-label-proxy/0.log"
Jul 21 18:51:04.696356 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:04.696330    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bc74c496-s4xs8_94326f12-be7e-45fc-a6c3-8e1709c96b02/kube-rbac-proxy-rules/0.log"
Jul 21 18:51:04.718982 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:04.718946    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68bc74c496-s4xs8_94326f12-be7e-45fc-a6c3-8e1709c96b02/kube-rbac-proxy-metrics/0.log"
Jul 21 18:51:06.626046 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:06.626012    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/2.log"
Jul 21 18:51:06.632888 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:06.632864    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7vppx_2302a4fe-a2b9-4de0-8dce-2805b3c2d825/console-operator/3.log"
Jul 21 18:51:07.645763 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.645735    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-8hdhq_52802b89-ae19-49fa-bc8c-6bb23a60b455/volume-data-source-validator/0.log"
Jul 21 18:51:07.884200 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.884172    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"]
Jul 21 18:51:07.884492 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.884480    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e2d0184-e683-47ab-b74e-b350662b0eba" containerName="storage-initializer"
Jul 21 18:51:07.884541 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.884494    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e2d0184-e683-47ab-b74e-b350662b0eba" containerName="storage-initializer"
Jul 21 18:51:07.884541 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.884505    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kube-rbac-proxy"
Jul 21 18:51:07.884541 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.884510    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kube-rbac-proxy"
Jul 21 18:51:07.884541 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.884518    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kserve-container"
Jul 21 18:51:07.884541 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.884524    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kserve-container"
Jul 21 18:51:07.884541 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.884536    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="storage-initializer"
Jul 21 18:51:07.884541 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.884542    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="storage-initializer"
Jul 21 18:51:07.884756 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.884595    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e2d0184-e683-47ab-b74e-b350662b0eba" containerName="storage-initializer"
Jul 21 18:51:07.884756 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.884602    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kserve-container"
Jul 21 18:51:07.884756 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.884608    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f651d6af-74ac-4c16-af46-5c8d86a36233" containerName="kube-rbac-proxy"
Jul 21 18:51:07.888764 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.888748    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:07.891241 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.891211    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4764r\"/\"kube-root-ca.crt\""
Jul 21 18:51:07.891384 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.891255    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4764r\"/\"openshift-service-ca.crt\""
Jul 21 18:51:07.892349 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.892332    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4764r\"/\"default-dockercfg-7dvvm\""
Jul 21 18:51:07.896920 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.896873    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"]
Jul 21 18:51:07.938907 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.938884    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2njqm\" (UniqueName: \"kubernetes.io/projected/6eaea295-7093-45e0-ab3e-bc1e1ab52cc6-kube-api-access-2njqm\") pod \"perf-node-gather-daemonset-gdrsz\" (UID: \"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:07.939007 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.938918    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6eaea295-7093-45e0-ab3e-bc1e1ab52cc6-sys\") pod \"perf-node-gather-daemonset-gdrsz\" (UID: \"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:07.939007 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.938937    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6eaea295-7093-45e0-ab3e-bc1e1ab52cc6-proc\") pod \"perf-node-gather-daemonset-gdrsz\" (UID: \"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:07.939007 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.938978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6eaea295-7093-45e0-ab3e-bc1e1ab52cc6-lib-modules\") pod \"perf-node-gather-daemonset-gdrsz\" (UID: \"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:07.939209 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:07.939037    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6eaea295-7093-45e0-ab3e-bc1e1ab52cc6-podres\") pod \"perf-node-gather-daemonset-gdrsz\" (UID: \"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:08.039601 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.039578    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2njqm\" (UniqueName: \"kubernetes.io/projected/6eaea295-7093-45e0-ab3e-bc1e1ab52cc6-kube-api-access-2njqm\") pod \"perf-node-gather-daemonset-gdrsz\" (UID: \"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:08.039741 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.039613    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6eaea295-7093-45e0-ab3e-bc1e1ab52cc6-sys\") pod \"perf-node-gather-daemonset-gdrsz\" (UID: \"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:08.039741 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.039631    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6eaea295-7093-45e0-ab3e-bc1e1ab52cc6-proc\") pod \"perf-node-gather-daemonset-gdrsz\" (UID: \"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:08.039741 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.039659    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6eaea295-7093-45e0-ab3e-bc1e1ab52cc6-lib-modules\") pod \"perf-node-gather-daemonset-gdrsz\" (UID: \"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:08.039741 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.039691    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6eaea295-7093-45e0-ab3e-bc1e1ab52cc6-podres\") pod \"perf-node-gather-daemonset-gdrsz\" (UID: \"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:08.039741 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.039715    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6eaea295-7093-45e0-ab3e-bc1e1ab52cc6-sys\") pod \"perf-node-gather-daemonset-gdrsz\" (UID: \"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:08.039929 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.039721    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6eaea295-7093-45e0-ab3e-bc1e1ab52cc6-proc\") pod \"perf-node-gather-daemonset-gdrsz\" (UID: \"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:08.039929 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.039804    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6eaea295-7093-45e0-ab3e-bc1e1ab52cc6-lib-modules\") pod \"perf-node-gather-daemonset-gdrsz\" (UID: \"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:08.039929 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.039836    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6eaea295-7093-45e0-ab3e-bc1e1ab52cc6-podres\") pod \"perf-node-gather-daemonset-gdrsz\" (UID: \"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:08.048103 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.048083    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2njqm\" (UniqueName: \"kubernetes.io/projected/6eaea295-7093-45e0-ab3e-bc1e1ab52cc6-kube-api-access-2njqm\") pod \"perf-node-gather-daemonset-gdrsz\" (UID: \"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:08.199331 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.199307    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:08.324857 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.324825    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"]
Jul 21 18:51:08.327871 ip-10-0-135-228 kubenswrapper[2562]: W0721 18:51:08.327846    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6eaea295_7093_45e0_ab3e_bc1e1ab52cc6.slice/crio-528aa2d8041a650cfd999512a7d0e816f8b9746b5d7a40809c9d6f332b76eaac WatchSource:0}: Error finding container 528aa2d8041a650cfd999512a7d0e816f8b9746b5d7a40809c9d6f332b76eaac: Status 404 returned error can't find the container with id 528aa2d8041a650cfd999512a7d0e816f8b9746b5d7a40809c9d6f332b76eaac
Jul 21 18:51:08.397287 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.397258    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz" event={"ID":"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6","Type":"ContainerStarted","Data":"6689d853b06c0c5205c106e3f3d754e8c71e6bc12532c9e8b0d6defbfd3bbc5a"}
Jul 21 18:51:08.397389 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.397311    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz" event={"ID":"6eaea295-7093-45e0-ab3e-bc1e1ab52cc6","Type":"ContainerStarted","Data":"528aa2d8041a650cfd999512a7d0e816f8b9746b5d7a40809c9d6f332b76eaac"}
Jul 21 18:51:08.397389 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.397341    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:08.415596 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.415554    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz" podStartSLOduration=1.415540531 podStartE2EDuration="1.415540531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 18:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:51:08.413087198 +0000 UTC m=+5119.984708972" watchObservedRunningTime="2026-07-21 18:51:08.415540531 +0000 UTC m=+5119.987162285"
Jul 21 18:51:08.591471 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.591381    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8zf7t_6ee3e5a6-186c-46a2-aef3-362ea08f9b30/dns/0.log"
Jul 21 18:51:08.612263 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.612238    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8zf7t_6ee3e5a6-186c-46a2-aef3-362ea08f9b30/kube-rbac-proxy/0.log"
Jul 21 18:51:08.760641 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:08.760618    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8gvpd_85fe9fd8-3fbd-4e4a-9300-b0c6c117fe15/dns-node-resolver/0.log"
Jul 21 18:51:09.313349 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:09.313272    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6bd858d955-9vzvr_627270a4-71a1-4982-9798-e3d68f23eddc/registry/0.log"
Jul 21 18:51:09.397207 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:09.397183    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xxbb8_b35c0345-124e-40c5-8de6-2b9c3122e24a/node-ca/0.log"
Jul 21 18:51:10.612898 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:10.612866    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r9btt_215830f3-8192-4423-a3fa-269ea3625142/serve-healthcheck-canary/0.log"
Jul 21 18:51:10.989535 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:10.989500    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-9qvnl_0a37a390-191f-4644-b398-6dcf97268ee8/insights-operator/0.log"
Jul 21 18:51:10.993981 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:10.993961    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-9qvnl_0a37a390-191f-4644-b398-6dcf97268ee8/insights-operator/1.log"
Jul 21 18:51:11.014749 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:11.014703    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6btrk_5757dbc1-e93d-4fcd-b888-c3e7cb4955b2/kube-rbac-proxy/0.log"
Jul 21 18:51:11.036949 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:11.036926    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6btrk_5757dbc1-e93d-4fcd-b888-c3e7cb4955b2/exporter/0.log"
Jul 21 18:51:11.059129 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:11.059109    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6btrk_5757dbc1-e93d-4fcd-b888-c3e7cb4955b2/extractor/0.log"
Jul 21 18:51:13.465336 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:13.465305    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-jvmmm_dd58785d-8d50-4068-a534-17aefa8fafd5/manager/0.log"
Jul 21 18:51:13.485744 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:13.485717    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-6cvvv_13722b77-a492-41cf-a54b-ca8845acc44e/server/0.log"
Jul 21 18:51:14.002818 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:14.002778    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-lnjkt_beba3329-ac69-4168-a5cf-7e171d9f8da5/manager/0.log"
Jul 21 18:51:14.233348 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:14.233308    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-z25x7_35e0b1cb-3151-4661-84df-9b281ea57b38/seaweedfs-tls-serving/0.log"
Jul 21 18:51:14.412256 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:14.412230    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4764r/perf-node-gather-daemonset-gdrsz"
Jul 21 18:51:18.800095 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:18.799774    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-56c8n_813ad497-8605-4f33-b614-32c185d701fc/kube-storage-version-migrator-operator/1.log"
Jul 21 18:51:18.801871 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:18.801841    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-56c8n_813ad497-8605-4f33-b614-32c185d701fc/kube-storage-version-migrator-operator/0.log"
Jul 21 18:51:19.939881 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:19.939855    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rnxt_7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1/kube-multus-additional-cni-plugins/0.log"
Jul 21 18:51:19.961110 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:19.961083    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rnxt_7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1/egress-router-binary-copy/0.log"
Jul 21 18:51:19.983234 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:19.983210    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rnxt_7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1/cni-plugins/0.log"
Jul 21 18:51:20.004312 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:20.004288    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rnxt_7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1/bond-cni-plugin/0.log"
Jul 21 18:51:20.025958 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:20.025931    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rnxt_7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1/routeoverride-cni/0.log"
Jul 21 18:51:20.047192 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:20.047171    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rnxt_7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1/whereabouts-cni-bincopy/0.log"
Jul 21 18:51:20.067677 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:20.067657    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rnxt_7fe18fe3-d1e6-4f34-a6b7-c1db7bafc4f1/whereabouts-cni/0.log"
Jul 21 18:51:20.310012 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:20.309930    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-tx58r_5e623c85-720f-4764-9898-3a0ecc93a45a/kube-multus/0.log"
Jul 21 18:51:20.363993 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:20.363963    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9r9z4_cb83c0b0-6020-43bf-be65-b41d51b76e5a/network-metrics-daemon/0.log"
Jul 21 18:51:20.382078 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:20.382041    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9r9z4_cb83c0b0-6020-43bf-be65-b41d51b76e5a/kube-rbac-proxy/0.log"
Jul 21 18:51:21.265011 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:21.264985    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdplm_0c36d72b-9bf6-4d72-a2b8-84451278a3d2/ovn-controller/0.log"
Jul 21 18:51:21.282584 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:21.282558    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdplm_0c36d72b-9bf6-4d72-a2b8-84451278a3d2/ovn-acl-logging/0.log"
Jul 21 18:51:21.310265 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:21.310242    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdplm_0c36d72b-9bf6-4d72-a2b8-84451278a3d2/kube-rbac-proxy-node/0.log"
Jul 21 18:51:21.333058 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:21.333036    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdplm_0c36d72b-9bf6-4d72-a2b8-84451278a3d2/kube-rbac-proxy-ovn-metrics/0.log"
Jul 21 18:51:21.351640 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:21.351618    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdplm_0c36d72b-9bf6-4d72-a2b8-84451278a3d2/northd/0.log"
Jul 21 18:51:21.372839 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:21.372819    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdplm_0c36d72b-9bf6-4d72-a2b8-84451278a3d2/nbdb/0.log"
Jul 21 18:51:21.394247 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:21.394214    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdplm_0c36d72b-9bf6-4d72-a2b8-84451278a3d2/sbdb/0.log"
Jul 21 18:51:21.468832 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:21.468800    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdplm_0c36d72b-9bf6-4d72-a2b8-84451278a3d2/ovnkube-controller/0.log"
Jul 21 18:51:23.027483 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:23.027449    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-r6qp2_c12e1bdf-b386-4a13-b3aa-a2a8b0ea4ab4/network-check-target-container/0.log"
Jul 21 18:51:23.984987 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:23.984950    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-5gtz4_a016f87b-eabf-47b0-a16f-94dafd80e108/iptables-alerter/0.log"
Jul 21 18:51:24.730087 ip-10-0-135-228 kubenswrapper[2562]: I0721 18:51:24.730039    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-rkbgr_4d4cddb0-f0f9-45b2-a606-00eca147717a/tuned/0.log"