Aug 04 18:00:00.593638 ip-10-0-132-242 systemd[1]: Starting Kubernetes Kubelet...
Aug 04 18:00:00.985468 ip-10-0-132-242 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 18:00:00.985468 ip-10-0-132-242 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Aug 04 18:00:00.985468 ip-10-0-132-242 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 18:00:00.985468 ip-10-0-132-242 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Aug 04 18:00:00.985468 ip-10-0-132-242 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 18:00:00.986156 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.985579    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Aug 04 18:00:00.991790 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991768    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:00.991790 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991786    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:00.991790 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991791    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991794    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991799    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991804    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991807    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991811    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991814    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991817    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991822    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991825    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991829    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991831    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991840    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991843    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991845    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991848    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991850    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991853    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991856    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:00.991891 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991858    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991861    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991863    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991866    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991868    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991871    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991873    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991876    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991879    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991881    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991884    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991886    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991889    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991891    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991894    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991897    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991900    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991903    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991906    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991909    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:00.992348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991913    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991916    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991919    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991922    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991924    2566 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991927    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991929    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991932    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991934    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991937    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991940    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991942    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991945    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991947    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991950    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991953    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991956    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991958    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991961    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991963    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:00.992897 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991966    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991969    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991971    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991974    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991977    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991979    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991982    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991984    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991987    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991990    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991993    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.991997    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992000    2566 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992002    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992005    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992008    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992010    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992013    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992016    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992018    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:00.993396 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992023    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992025    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992028    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992030    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992033    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992036    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992040    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992043    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992045    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992048    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992051    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992053    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992055    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992058    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992061    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992063    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992066    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.992068    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993384    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993392    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:00.993886 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993396    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993399    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993402    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993407    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993410    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993413    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993416    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993420    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993424    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993427    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993430    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993434    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993437    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993439    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993443    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993445    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993448    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993451    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993454    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:00.994390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993456    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993459    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993462    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993464    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993467    2566 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993470    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993473    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993475    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993478    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993480    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993483    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993485    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993487    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993490    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993493    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993496    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993499    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993509    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993512    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993515    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:00.994879 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993517    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993521    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993523    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993526    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993528    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993531    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993534    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993536    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993539    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993541    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993544    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993547    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993550    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993553    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993555    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993559    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993562    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993565    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993568    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993570    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:00.995467 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993573    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993576    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993579    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993583    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993587    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993590    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993592    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993595    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993604    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993607    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993615    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993618    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993621    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993623    2566 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993626    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993628    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993631    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993634    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993636    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:00.995951 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993639    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993642    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993644    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993647    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993649    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993652    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993654    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993657    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993659    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993662    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993664    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993666    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993669    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993671    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993674    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993677    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993679    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993681    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.993684    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994095    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:00.996439 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994101    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994105    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994107    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994112    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994115    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994118    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994120    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994123    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994126    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994130    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994134    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994137    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994139    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994142    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994144    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994147    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994150    2566 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994153    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994155    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:00.996930 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994158    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994160    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994163    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994166    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994168    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994170    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994173    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994175    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994178    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994180    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994183    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994185    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994187    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994191    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994194    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994196    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994199    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994202    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994205    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:00.997416 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994207    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994210    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994212    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994216    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994218    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994222    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994226    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994228    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994231    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994233    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994236    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994238    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994241    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994244    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994246    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994249    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994251    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994254    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994256    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:00.997867 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994259    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994261    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994264    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994266    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994269    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994271    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994274    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994277    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994279    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994282    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994285    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994287    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994290    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994293    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994295    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994298    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994301    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994303    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994306    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994308    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:00.998329 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994310    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994313    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994315    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994318    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994320    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994323    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994326    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994328    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994331    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994335    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994338    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994341    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994343    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994346    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994365    2566 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994369    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994372    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994375    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994377    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:00.998856 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994380    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.994383    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994455    2566 flags.go:64] FLAG: --address="0.0.0.0"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994466    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994477    2566 flags.go:64] FLAG: --anonymous-auth="true"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994483    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994490    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994495    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994502    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994509    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994512    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994515    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994518    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994521    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994524    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994527    2566 flags.go:64] FLAG: --cgroup-root=""
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994530    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994533    2566 flags.go:64] FLAG: --client-ca-file=""
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994538    2566 flags.go:64] FLAG: --cloud-provider="external"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994542    2566 flags.go:64] FLAG: --cluster-dns="[]"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994547    2566 flags.go:64] FLAG: --cluster-domain=""
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994550    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994553    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994556    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 04 18:00:00.999328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994560    2566 flags.go:64] FLAG: --container-log-max-files="5"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994563    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994566    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994569    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994573    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994575    2566 flags.go:64] FLAG: --contention-profiling="false"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994578    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994582    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994585    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994588    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994592    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994595    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994598    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994601    2566 flags.go:64] FLAG: --enable-load-reader="false"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994604    2566 flags.go:64] FLAG: --enable-server="true"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994608    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994617    2566 flags.go:64] FLAG: --event-burst="100"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994620    2566 flags.go:64] FLAG: --event-qps="50"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994623    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994626    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994629    2566 flags.go:64] FLAG: --eviction-hard=""
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994633    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994636    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994639    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994641    2566 flags.go:64] FLAG: --eviction-soft=""
Aug 04 18:00:00.999927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994644    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994649    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994652    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994655    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994658    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994661    2566 flags.go:64] FLAG: --fail-swap-on="true"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994664    2566 flags.go:64] FLAG: --feature-gates=""
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994668    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994671    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994674    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994677    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994680    2566 flags.go:64] FLAG: --healthz-port="10248"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994684    2566 flags.go:64] FLAG: --help="false"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994687    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994689    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994693    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994696    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994699    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994703    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994706    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994709    2566 flags.go:64] FLAG: --image-service-endpoint=""
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994712    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994715    2566 flags.go:64] FLAG: --kube-api-burst="100"
Aug 04 18:00:01.000636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994720    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994723    2566 flags.go:64] FLAG: --kube-api-qps="50"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994733    2566 flags.go:64] FLAG: --kube-reserved=""
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994737    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994740    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994743    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994746    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994749    2566 flags.go:64] FLAG: --lock-file=""
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994752    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994755    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994760    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994765    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994768    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994771    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994774    2566 flags.go:64] FLAG: --logging-format="text"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994778    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994781    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994784    2566 flags.go:64] FLAG: --manifest-url=""
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994787    2566 flags.go:64] FLAG: --manifest-url-header=""
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994792    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994795    2566 flags.go:64] FLAG: --max-open-files="1000000"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994799    2566 flags.go:64] FLAG: --max-pods="110"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994802    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994806    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994809    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Aug 04 18:00:01.001551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994812    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994815    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994818    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994821    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994829    2566 flags.go:64] FLAG: --node-status-max-images="50"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994832    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994835    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994840    2566 flags.go:64] FLAG: --pod-cidr=""
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994843    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994849    2566 flags.go:64] FLAG: --pod-manifest-path=""
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994852    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994855    2566 flags.go:64] FLAG: --pods-per-core="0"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994858    2566 flags.go:64] FLAG: --port="10250"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994861    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994864    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09164315e3bdeaa81"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994867    2566 flags.go:64] FLAG: --qos-reserved=""
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994870    2566 flags.go:64] FLAG: --read-only-port="10255"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994874    2566 flags.go:64] FLAG: --register-node="true"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994877    2566 flags.go:64] FLAG: --register-with-taints=""
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994881    2566 flags.go:64] FLAG: --registry-burst="10"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994884    2566 flags.go:64] FLAG: --registry-qps="5"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994887    2566 flags.go:64] FLAG: --reserved-cpus=""
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994890    2566 flags.go:64] FLAG: --reserved-memory=""
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994898    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994901    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Aug 04 18:00:01.002169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994904    2566 flags.go:64] FLAG: --rotate-certificates="false"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994907    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994910    2566 flags.go:64] FLAG: --runonce="false"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994913    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994917    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994919    2566 flags.go:64] FLAG: --seccomp-default="false"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994923    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994926    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994929    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994932    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994935    2566 flags.go:64] FLAG: --storage-driver-password="root"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994938    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994941    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994944    2566 flags.go:64] FLAG: --storage-driver-user="root"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994947    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994951    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994955    2566 flags.go:64] FLAG: --system-cgroups=""
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994958    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994963    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994966    2566 flags.go:64] FLAG: --tls-cert-file=""
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994969    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994974    2566 flags.go:64] FLAG: --tls-min-version=""
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994976    2566 flags.go:64] FLAG: --tls-private-key-file=""
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994979    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994983    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Aug 04 18:00:01.002823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994986    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994989    2566 flags.go:64] FLAG: --v="2"
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994994    2566 flags.go:64] FLAG: --version="false"
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.994998    2566 flags.go:64] FLAG: --vmodule=""
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.995002    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.995005    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995134    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995137    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995141    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995143    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995146    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995148    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995151    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995154    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995157    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995159    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995162    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995164    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995167    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995170    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995172    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:01.003460 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995175    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995179    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995182    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995185    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995188    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995190    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995193    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995195    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995198    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995201    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995205    2566 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995208    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995210    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995213    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995215    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995218    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995220    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995223    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995225    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995228    2566 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:01.003987 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995230    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995233    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995236    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995240    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995244    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995247    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995250    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995253    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995257    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995260    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995263    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995266    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995268    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995272    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995275    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995277    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995281    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995283    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995286    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:01.004532 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995289    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995291    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995294    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995298    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995300    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995303    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995305    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995309    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995311    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995314    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995317    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995320    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995322    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995325    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995327    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995330    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995332    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995335    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995337    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995340    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:01.005010 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995342    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995346    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995362    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995366    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995369    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995371    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995376    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995379    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995381    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995384    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995394    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995397    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995399    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995402    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995404    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995408    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995411    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995413    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995416    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:01.005518 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995419    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:01.005990 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995421    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:01.005990 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995424    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:01.005990 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995427    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:01.005990 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995429    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:01.005990 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:00.995432    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:01.005990 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:00.995437    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 18:00:01.005990 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.002635    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Aug 04 18:00:01.005990 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.002652    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Aug 04 18:00:01.005990 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.002672    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Aug 04 18:00:01.005990 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.002678    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Aug 04 18:00:01.005990 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002762    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:01.005990 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002768    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:01.005990 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002771    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:01.005990 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002774    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:01.005990 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002777    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002780    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002783    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002786    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002788    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002791    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002793    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002796    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002799    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002802    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002805    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002808    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002810    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002813    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002815    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002818    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002821    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002823    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002826    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:01.006379 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002829    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002831    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002834    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002837    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002840    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002843    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002846    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002849    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002852    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002854    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002857    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002860    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002862    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002864    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002867    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002869    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002872    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002876    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002880    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002883    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:01.006862 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002886    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002888    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002892    2566 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002895    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002898    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002900    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002903    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002905    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002908    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002911    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002913    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002915    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002918    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002920    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002924    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002926    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002929    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002931    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002934    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002937    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:01.007346 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002940    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002943    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002945    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002948    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002951    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002953    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002956    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002958    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002961    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002963    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002966    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002968    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002971    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002973    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002976    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002979    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002981    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002985    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002989    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:01.007906 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002991    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:01.008780 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002994    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:01.008780 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.002999    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:01.008780 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003002    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:01.008780 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003005    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:01.008780 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003008    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:01.008780 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003012    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:01.008780 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003015    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:01.008780 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003017    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:01.008780 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003020    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:01.008780 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003023    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:01.008780 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003026    2566 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:01.008780 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003028    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:01.008780 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003030    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:01.008780 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003033    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:01.008780 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003035    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:01.008780 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003038    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:01.009427 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.003043    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 18:00:01.009427 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003149    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:01.009427 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003153    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:01.009427 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003156    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:01.009427 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003159    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:01.009427 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003162    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:01.009427 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003165    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:01.009427 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003168    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:01.009427 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003170    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:01.009427 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003173    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:01.009427 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003175    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:01.009427 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003178    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:01.009427 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003182    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:01.009427 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003184    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:01.009427 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003187    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003190    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003194    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003196    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003199    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003203    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003206    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003209    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003212    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003215    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003218    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003221    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003224    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003226    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003229    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003232    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003234    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003237    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:01.009922 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003239    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003242    2566 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003244    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003246    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003249    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003251    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003254    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003256    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003259    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003261    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003264    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003266    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003269    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003272    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003275    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003277    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003279    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003282    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003284    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003287    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:01.010758 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003289    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003292    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003295    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003297    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003300    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003302    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003305    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003307    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003310    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003312    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003315    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003317    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003320    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003322    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003325    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003327    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003329    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003332    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003334    2566 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003337    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:01.011621 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003339    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003342    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003344    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003347    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003363    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003366    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003369    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003371    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003374    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003377    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003379    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003382    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003384    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003387    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003391    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003394    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003398    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003401    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003403    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003406    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:01.012457 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003409    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:01.012955 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003412    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:01.012955 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003414    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:01.012955 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003417    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:01.012955 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003420    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:01.012955 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003423    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:01.012955 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003426    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:01.012955 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:01.003428    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:01.012955 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.003433    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 18:00:01.012955 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.003556    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Aug 04 18:00:01.012955 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.005548    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Aug 04 18:00:01.012955 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.006863    2566 server.go:1019] "Starting client certificate rotation"
Aug 04 18:00:01.012955 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.006964    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 18:00:01.012955 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.007003    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 18:00:01.013304 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.010567    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 18:00:01.013304 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.012376    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 18:00:01.023031 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.023009    2566 log.go:25] "Validated CRI v1 runtime API"
Aug 04 18:00:01.025225 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.025208    2566 log.go:25] "Validated CRI v1 image API"
Aug 04 18:00:01.025976 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.025962    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Aug 04 18:00:01.028001 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.027980    2566 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7fa708e0-5fd7-4182-ac76-ff75d6538081:/dev/nvme0n1p4 fe82ffac-0aed-403f-bd24-51302a323de8:/dev/nvme0n1p3]
Aug 04 18:00:01.028076 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.028000    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Aug 04 18:00:01.034386 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.034264    2566 manager.go:216] Machine: {Timestamp:2026-08-04 18:00:01.032962769 +0000 UTC m=+0.323576952 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3079136 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e66eea72018b38baac7cecfab7be2 SystemUUID:ec2e66ee-a720-18b3-8baa-c7cecfab7be2 BootID:ec7856d3-90d1-458f-93d7-94e9ab1b87f1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:11:c3:69:45:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:11:c3:69:45:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:ee:8c:a4:db:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 04 18:00:01.034386 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.034381    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Aug 04 18:00:01.034517 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.034505    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Aug 04 18:00:01.037674 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.037658    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 18:00:01.037754 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.037729    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Aug 04 18:00:01.037902 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.037759    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-242.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Aug 04 18:00:01.038389 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.038378    2566 topology_manager.go:138] "Creating topology manager with none policy"
Aug 04 18:00:01.038420 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.038391    2566 container_manager_linux.go:309] "Creating device plugin manager"
Aug 04 18:00:01.038420 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.038399    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 18:00:01.039416 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.039404    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 18:00:01.040584 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.040574    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Aug 04 18:00:01.044198 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.044185    2566 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 18:00:01.044325 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.044316    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Aug 04 18:00:01.044374 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.044345    2566 kubelet.go:486] "Attempting to sync node with API server"
Aug 04 18:00:01.044374 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.044370    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Aug 04 18:00:01.044435 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.044383    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Aug 04 18:00:01.044435 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.044392    2566 kubelet.go:398] "Adding apiserver pod source"
Aug 04 18:00:01.044435 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.044401    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Aug 04 18:00:01.044779 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.044760    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 18:00:01.044828 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.044795    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 18:00:01.047364 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.047328    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Aug 04 18:00:01.050531 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.050511    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Aug 04 18:00:01.050592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.050559    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Aug 04 18:00:01.051961 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.051937    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Aug 04 18:00:01.052030 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.051982    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Aug 04 18:00:01.052030 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.051989    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Aug 04 18:00:01.052030 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.051996    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Aug 04 18:00:01.052030 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.052005    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Aug 04 18:00:01.052030 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.052012    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Aug 04 18:00:01.052030 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.052021    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Aug 04 18:00:01.052030 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.052027    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Aug 04 18:00:01.052260 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.052035    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Aug 04 18:00:01.052260 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.052045    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Aug 04 18:00:01.052260 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.052100    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Aug 04 18:00:01.052260 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.052258    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Aug 04 18:00:01.053061 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.053035    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Aug 04 18:00:01.053275 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.053065    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Aug 04 18:00:01.055901 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.055881    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-242.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Aug 04 18:00:01.056750 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.056728    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Aug 04 18:00:01.056750 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.056730    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Aug 04 18:00:01.056883 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.056800    2566 server.go:1268] "Started kubelet"
Aug 04 18:00:01.056937 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.056875    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Aug 04 18:00:01.057021 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.056953    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Aug 04 18:00:01.057064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.057046    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Aug 04 18:00:01.057844 ip-10-0-132-242 systemd[1]: Started Kubernetes Kubelet.
Aug 04 18:00:01.058167 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.058152    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Aug 04 18:00:01.059363 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.059338    2566 server.go:310] "Adding debug handlers to kubelet server"
Aug 04 18:00:01.063005 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.062986    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Aug 04 18:00:01.063136 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.063114    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bwhfw"
Aug 04 18:00:01.063196 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.063157    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Aug 04 18:00:01.063457 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.063415    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-242.ec2.internal\" not found"
Aug 04 18:00:01.063457 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.063428    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Aug 04 18:00:01.063858 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.063840    2566 factory.go:55] Registering systemd factory
Aug 04 18:00:01.063945 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.063877    2566 factory.go:223] Registration of the systemd container factory successfully
Aug 04 18:00:01.064004 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.063979    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Aug 04 18:00:01.064004 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.063981    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Aug 04 18:00:01.064099 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.064009    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Aug 04 18:00:01.064099 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.064090    2566 factory.go:153] Registering CRI-O factory
Aug 04 18:00:01.064180 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.064106    2566 factory.go:223] Registration of the crio container factory successfully
Aug 04 18:00:01.064180 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.064108    2566 reconstruct.go:97] "Volume reconstruction finished"
Aug 04 18:00:01.064180 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.064117    2566 reconciler.go:29] "Reconciler: start to sync state"
Aug 04 18:00:01.064180 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.064169    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Aug 04 18:00:01.064396 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.064192    2566 factory.go:103] Registering Raw factory
Aug 04 18:00:01.064396 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.064207    2566 manager.go:1196] Started watching for new ooms in manager
Aug 04 18:00:01.064647 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.064632    2566 manager.go:349] Starting recovery of all containers
Aug 04 18:00:01.065826 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.065804    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Aug 04 18:00:01.065907 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.065837    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Aug 04 18:00:01.066920 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.065875    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-242.ec2.internal.18c8ac07a34b4fd3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-242.ec2.internal,UID:ip-10-0-132-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-242.ec2.internal,},FirstTimestamp:2026-08-04 18:00:01.056772051 +0000 UTC m=+0.347386216,LastTimestamp:2026-08-04 18:00:01.056772051 +0000 UTC m=+0.347386216,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-242.ec2.internal,}"
Aug 04 18:00:01.070283 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.070258    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bwhfw"
Aug 04 18:00:01.074754 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.074736    2566 manager.go:354] Recovery completed
Aug 04 18:00:01.081480 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.081464    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:01.084041 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.084017    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:01.084135 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.084053    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:01.084135 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.084069    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:01.085496 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.085474    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Aug 04 18:00:01.085496 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.085487    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Aug 04 18:00:01.085625 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.085506    2566 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 18:00:01.085625 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.085590    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:01.086328 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.086259    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-242.ec2.internal.18c8ac07a4eb57d9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-242.ec2.internal,UID:ip-10-0-132-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-242.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-242.ec2.internal,},FirstTimestamp:2026-08-04 18:00:01.084037081 +0000 UTC m=+0.374651248,LastTimestamp:2026-08-04 18:00:01.084037081 +0000 UTC m=+0.374651248,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-242.ec2.internal,}"
Aug 04 18:00:01.086617 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.086604    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:01.086667 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.086630    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:01.086667 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.086640    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:01.086746 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.086724    2566 policy_none.go:49] "None policy: Start"
Aug 04 18:00:01.086780 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.086748    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Aug 04 18:00:01.086780 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.086759    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Aug 04 18:00:01.087657 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.087644    2566 policy_none.go:47] "Start"
Aug 04 18:00:01.125824 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.125634    2566 manager.go:337] "Starting Device Plugin manager"
Aug 04 18:00:01.125933 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.125849    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Aug 04 18:00:01.125933 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.125862    2566 server.go:87] "Starting device plugin registration server"
Aug 04 18:00:01.126169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.126157    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 04 18:00:01.126221 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.126170    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 04 18:00:01.126295 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.126271    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Aug 04 18:00:01.126396 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.126374    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Aug 04 18:00:01.126396 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.126385    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 04 18:00:01.126908 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.126882    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Aug 04 18:00:01.126999 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.126944    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-242.ec2.internal\" not found"
Aug 04 18:00:01.187113 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.187079    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Aug 04 18:00:01.188215 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.188201    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Aug 04 18:00:01.188290 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.188243    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Aug 04 18:00:01.188290 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.188271    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Aug 04 18:00:01.188403 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.188313    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Aug 04 18:00:01.190772 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.190753    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:01.226967 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.226935    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:01.229380 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.229364    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:01.229454 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.229392    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:01.229454 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.229402    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:01.229454 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.229427    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.237663 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.237591    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.237663 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.237615    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-242.ec2.internal\": node \"ip-10-0-132-242.ec2.internal\" not found"
Aug 04 18:00:01.252623 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.252596    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-242.ec2.internal\" not found"
Aug 04 18:00:01.288417 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.288373    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-242.ec2.internal"]
Aug 04 18:00:01.288535 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.288466    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:01.290874 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.290860    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:01.290937 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.290890    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:01.290937 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.290901    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:01.292222 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.292209    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:01.292353 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.292338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.292398 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.292388    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:01.293052 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.293038    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:01.293119 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.293063    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:01.293119 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.293067    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:01.293119 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.293079    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:01.293119 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.293090    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:01.293119 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.293101    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:01.294309 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.294294    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.294356 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.294324    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:01.295106 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.295087    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:01.295195 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.295141    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:01.295195 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.295157    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:01.322267 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.322242    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-242.ec2.internal\" not found" node="ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.327251 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.327234    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-242.ec2.internal\" not found" node="ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.352996 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.352973    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-242.ec2.internal\" not found"
Aug 04 18:00:01.365572 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.365543    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d7cee5e53dc47fb7193759f38d87999-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal\" (UID: \"3d7cee5e53dc47fb7193759f38d87999\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.365667 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.365574    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d7cee5e53dc47fb7193759f38d87999-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal\" (UID: \"3d7cee5e53dc47fb7193759f38d87999\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.365667 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.365592    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b78a122ed0f6c6c547f0087e492e65b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-242.ec2.internal\" (UID: \"4b78a122ed0f6c6c547f0087e492e65b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.453895 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.453852    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-242.ec2.internal\" not found"
Aug 04 18:00:01.466073 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.466048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d7cee5e53dc47fb7193759f38d87999-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal\" (UID: \"3d7cee5e53dc47fb7193759f38d87999\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.466159 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.466082    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d7cee5e53dc47fb7193759f38d87999-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal\" (UID: \"3d7cee5e53dc47fb7193759f38d87999\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.466159 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.466112    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b78a122ed0f6c6c547f0087e492e65b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-242.ec2.internal\" (UID: \"4b78a122ed0f6c6c547f0087e492e65b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.466245 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.466164    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b78a122ed0f6c6c547f0087e492e65b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-242.ec2.internal\" (UID: \"4b78a122ed0f6c6c547f0087e492e65b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.466245 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.466179    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d7cee5e53dc47fb7193759f38d87999-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal\" (UID: \"3d7cee5e53dc47fb7193759f38d87999\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.466245 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.466179    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d7cee5e53dc47fb7193759f38d87999-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal\" (UID: \"3d7cee5e53dc47fb7193759f38d87999\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.554176 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.554089    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-242.ec2.internal\" not found"
Aug 04 18:00:01.629251 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.629219    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.630416 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:01.630396    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-242.ec2.internal"
Aug 04 18:00:01.654965 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.654935    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-242.ec2.internal\" not found"
Aug 04 18:00:01.760080 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.760035    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-242.ec2.internal\" not found"
Aug 04 18:00:01.865910 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.865821    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-242.ec2.internal\" not found"
Aug 04 18:00:01.966955 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:01.966903    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-242.ec2.internal\" not found"
Aug 04 18:00:02.008367 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:02.008342    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Aug 04 18:00:02.008778 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:02.008532    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 04 18:00:02.067030 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:02.066997    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-242.ec2.internal\" not found"
Aug 04 18:00:02.074382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:02.074340    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 17:55:01 +0000 UTC" deadline="2028-04-10 20:39:54.814895594 +0000 UTC"
Aug 04 18:00:02.074382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:02.074379    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14762h39m52.740520721s"
Aug 04 18:00:02.155394 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:02.155338    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:02.167123 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:02.167093    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-242.ec2.internal\" not found"
Aug 04 18:00:02.269013 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:02.268964    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-242.ec2.internal\" not found"
Aug 04 18:00:02.370808 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:02.370785    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-242.ec2.internal\" not found"
Aug 04 18:00:02.471223 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:02.471149    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-242.ec2.internal\" not found"
Aug 04 18:00:02.472841 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:02.472816    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:02.553564 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:02.553533    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:02.564256 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:02.564235    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal"
Aug 04 18:00:02.576413 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:02.576386    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 18:00:02.577457 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:02.577442    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-242.ec2.internal"
Aug 04 18:00:02.587897 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:02.587878    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 18:00:03.046740 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.046710    2566 apiserver.go:52] "Watching apiserver"
Aug 04 18:00:03.055985 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.055702    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Aug 04 18:00:03.056691 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.056666    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-kfzm2","kube-system/kube-apiserver-proxy-ip-10-0-132-242.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal","openshift-multus/multus-82njr","openshift-multus/multus-additional-cni-plugins-hmzpt","openshift-multus/network-metrics-daemon-647mg","openshift-network-diagnostics/network-check-target-bd4m4","openshift-network-operator/iptables-alerter-tzdhp","kube-system/global-pull-secret-syncer-t46pw","kube-system/konnectivity-agent-wjjnh","openshift-cluster-node-tuning-operator/tuned-lfd9z","openshift-dns/node-resolver-v69qf","openshift-image-registry/node-ca-7f2bk"]
Aug 04 18:00:03.058269 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.058253    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:03.058345 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.058327    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:03.059298 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.059283    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v69qf"
Aug 04 18:00:03.060436 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.060413    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.061436 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.061416    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7f2bk"
Aug 04 18:00:03.061697 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.061669    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Aug 04 18:00:03.061797 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.061736    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Aug 04 18:00:03.061797 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.061793    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wj6rg\""
Aug 04 18:00:03.062530 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.062515    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Aug 04 18:00:03.062608 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.062584    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Aug 04 18:00:03.062608 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.062594    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.062608 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.062605    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Aug 04 18:00:03.063191 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.063177    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5qffw\""
Aug 04 18:00:03.063696 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.063683    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Aug 04 18:00:03.063775 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.063685    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Aug 04 18:00:03.063931 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.063917    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.064133 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.064116    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Aug 04 18:00:03.064222 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.064123    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Aug 04 18:00:03.064222 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.064213    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k7kdb\""
Aug 04 18:00:03.064943 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.064931    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Aug 04 18:00:03.065039 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.065021    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-48sr2\""
Aug 04 18:00:03.065088 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.065026    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tzdhp"
Aug 04 18:00:03.065131 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.065094    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Aug 04 18:00:03.065260 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.065244    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Aug 04 18:00:03.065410 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.065395    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Aug 04 18:00:03.065992 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.065973    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Aug 04 18:00:03.066106 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.066088    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Aug 04 18:00:03.066166 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.066093    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-slbq2\""
Aug 04 18:00:03.066166 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.066156    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:03.066266 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.066226    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:03.067282 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.067266    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:03.067427 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.067415    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:03.067488 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.067472    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:03.067579 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.067565    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:03.067631 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.067592    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Aug 04 18:00:03.067673 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.067594    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hgxqf\""
Aug 04 18:00:03.068751 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.068738    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.069982 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.069965    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wjjnh"
Aug 04 18:00:03.070812 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.070794    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Aug 04 18:00:03.071025 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.071010    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.071763 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.071745    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nq5cc\""
Aug 04 18:00:03.071860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.071765    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Aug 04 18:00:03.071860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.071777    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Aug 04 18:00:03.071860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.071797    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Aug 04 18:00:03.071860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.071805    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Aug 04 18:00:03.071860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.071771    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Aug 04 18:00:03.072148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.072135    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Aug 04 18:00:03.072200 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.072135    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Aug 04 18:00:03.072252 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.072239    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lkvd2\""
Aug 04 18:00:03.073055 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.073040    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:03.073102 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.073063    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:03.073307 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.073295    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bj8l9\""
Aug 04 18:00:03.075782 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.075766    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-run-netns\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.075860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.075788    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c68d\" (UniqueName: \"kubernetes.io/projected/883bdd22-2b82-4cd5-bc0b-28555ec88938-kube-api-access-9c68d\") pod \"iptables-alerter-tzdhp\" (UID: \"883bdd22-2b82-4cd5-bc0b-28555ec88938\") " pod="openshift-network-operator/iptables-alerter-tzdhp"
Aug 04 18:00:03.075860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.075803    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dce6b12-1695-48e9-b13a-0bf64d3107ff-serviceca\") pod \"node-ca-7f2bk\" (UID: \"7dce6b12-1695-48e9-b13a-0bf64d3107ff\") " pod="openshift-image-registry/node-ca-7f2bk"
Aug 04 18:00:03.075860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.075818    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2959820d-b173-49ec-875d-9dfa00f47f50-cni-binary-copy\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.075860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.075838    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6cbc3f49-e03c-425d-9da2-71eb15915d42-dbus\") pod \"global-pull-secret-syncer-t46pw\" (UID: \"6cbc3f49-e03c-425d-9da2-71eb15915d42\") " pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:03.075993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.075867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-cnibin\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.075993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.075896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-etc-kubernetes\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.075993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.075921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/883bdd22-2b82-4cd5-bc0b-28555ec88938-host-slash\") pod \"iptables-alerter-tzdhp\" (UID: \"883bdd22-2b82-4cd5-bc0b-28555ec88938\") " pod="openshift-network-operator/iptables-alerter-tzdhp"
Aug 04 18:00:03.075993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.075945    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-registration-dir\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.075993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.075967    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfx6r\" (UniqueName: \"kubernetes.io/projected/834b97ef-bbb4-44bd-a065-8408f4523f06-kube-api-access-gfx6r\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.075993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.075989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhs2x\" (UniqueName: \"kubernetes.io/projected/7dce6b12-1695-48e9-b13a-0bf64d3107ff-kube-api-access-jhs2x\") pod \"node-ca-7f2bk\" (UID: \"7dce6b12-1695-48e9-b13a-0bf64d3107ff\") " pod="openshift-image-registry/node-ca-7f2bk"
Aug 04 18:00:03.076160 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076018    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-os-release\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.076160 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-sys-fs\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.076160 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076058    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2959820d-b173-49ec-875d-9dfa00f47f50-system-cni-dir\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.076160 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076077    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2959820d-b173-49ec-875d-9dfa00f47f50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.076160 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076102    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret\") pod \"global-pull-secret-syncer-t46pw\" (UID: \"6cbc3f49-e03c-425d-9da2-71eb15915d42\") " pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:03.076160 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-multus-socket-dir-parent\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.076160 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-run-k8s-cni-cncf-io\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.076401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076167    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-hostroot\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.076401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076184    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/703eadd7-667a-4879-aa4e-2ccedb5b7c8a-tmp-dir\") pod \"node-resolver-v69qf\" (UID: \"703eadd7-667a-4879-aa4e-2ccedb5b7c8a\") " pod="openshift-dns/node-resolver-v69qf"
Aug 04 18:00:03.076401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-device-dir\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.076401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076225    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-etc-selinux\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.076401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076268    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-multus-cni-dir\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.076401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076294    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09c8e863-111d-467c-9c07-d0d6758b2e20-cni-binary-copy\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.076401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-run-multus-certs\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.076401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/883bdd22-2b82-4cd5-bc0b-28555ec88938-iptables-alerter-script\") pod \"iptables-alerter-tzdhp\" (UID: \"883bdd22-2b82-4cd5-bc0b-28555ec88938\") " pod="openshift-network-operator/iptables-alerter-tzdhp"
Aug 04 18:00:03.076401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076374    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/703eadd7-667a-4879-aa4e-2ccedb5b7c8a-hosts-file\") pod \"node-resolver-v69qf\" (UID: \"703eadd7-667a-4879-aa4e-2ccedb5b7c8a\") " pod="openshift-dns/node-resolver-v69qf"
Aug 04 18:00:03.076401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-socket-dir\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.076758 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2959820d-b173-49ec-875d-9dfa00f47f50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.076758 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-var-lib-cni-bin\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.076758 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076517    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-var-lib-cni-multus\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.076758 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076538    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.076758 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2959820d-b173-49ec-875d-9dfa00f47f50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.076758 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076584    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7cv6\" (UniqueName: \"kubernetes.io/projected/2959820d-b173-49ec-875d-9dfa00f47f50-kube-api-access-c7cv6\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.076758 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6cbc3f49-e03c-425d-9da2-71eb15915d42-kubelet-config\") pod \"global-pull-secret-syncer-t46pw\" (UID: \"6cbc3f49-e03c-425d-9da2-71eb15915d42\") " pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:03.076758 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-var-lib-kubelet\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.076758 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076687    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-multus-conf-dir\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.076758 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076709    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dce6b12-1695-48e9-b13a-0bf64d3107ff-host\") pod \"node-ca-7f2bk\" (UID: \"7dce6b12-1695-48e9-b13a-0bf64d3107ff\") " pod="openshift-image-registry/node-ca-7f2bk"
Aug 04 18:00:03.076758 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2959820d-b173-49ec-875d-9dfa00f47f50-cnibin\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.076758 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076752    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2959820d-b173-49ec-875d-9dfa00f47f50-os-release\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.077179 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-system-cni-dir\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.077179 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076800    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09c8e863-111d-467c-9c07-d0d6758b2e20-multus-daemon-config\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.077179 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g2x7\" (UniqueName: \"kubernetes.io/projected/09c8e863-111d-467c-9c07-d0d6758b2e20-kube-api-access-9g2x7\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.077179 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.076846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t95p\" (UniqueName: \"kubernetes.io/projected/703eadd7-667a-4879-aa4e-2ccedb5b7c8a-kube-api-access-5t95p\") pod \"node-resolver-v69qf\" (UID: \"703eadd7-667a-4879-aa4e-2ccedb5b7c8a\") " pod="openshift-dns/node-resolver-v69qf"
Aug 04 18:00:03.084841 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.084822    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 18:00:03.115200 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.115178    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2dcd5"
Aug 04 18:00:03.136856 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.136827    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2dcd5"
Aug 04 18:00:03.165176 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.165154    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 04 18:00:03.177143 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177027    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-run-systemd\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.177143 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177066    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-var-lib-openvswitch\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.177143 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-run-ovn\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.177143 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-multus-conf-dir\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.177346 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dce6b12-1695-48e9-b13a-0bf64d3107ff-host\") pod \"node-ca-7f2bk\" (UID: \"7dce6b12-1695-48e9-b13a-0bf64d3107ff\") " pod="openshift-image-registry/node-ca-7f2bk"
Aug 04 18:00:03.177346 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2959820d-b173-49ec-875d-9dfa00f47f50-os-release\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.177346 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177201    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-tuned\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.177346 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177220    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/460132b7-d6bd-44f3-b852-1de1afa84bf8-tmp\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.177346 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177240    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-etc-openvswitch\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.177346 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.177346 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177311    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-system-cni-dir\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.177346 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177337    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09c8e863-111d-467c-9c07-d0d6758b2e20-multus-daemon-config\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.177744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177378    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-slash\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.177744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177404    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c68d\" (UniqueName: \"kubernetes.io/projected/883bdd22-2b82-4cd5-bc0b-28555ec88938-kube-api-access-9c68d\") pod \"iptables-alerter-tzdhp\" (UID: \"883bdd22-2b82-4cd5-bc0b-28555ec88938\") " pod="openshift-network-operator/iptables-alerter-tzdhp"
Aug 04 18:00:03.177744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177429    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dce6b12-1695-48e9-b13a-0bf64d3107ff-serviceca\") pod \"node-ca-7f2bk\" (UID: \"7dce6b12-1695-48e9-b13a-0bf64d3107ff\") " pod="openshift-image-registry/node-ca-7f2bk"
Aug 04 18:00:03.177744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177456    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2959820d-b173-49ec-875d-9dfa00f47f50-cni-binary-copy\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.177744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177468    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dce6b12-1695-48e9-b13a-0bf64d3107ff-host\") pod \"node-ca-7f2bk\" (UID: \"7dce6b12-1695-48e9-b13a-0bf64d3107ff\") " pod="openshift-image-registry/node-ca-7f2bk"
Aug 04 18:00:03.177744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177479    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-host\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.177744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177566    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-cnibin\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.177744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/883bdd22-2b82-4cd5-bc0b-28555ec88938-host-slash\") pod \"iptables-alerter-tzdhp\" (UID: \"883bdd22-2b82-4cd5-bc0b-28555ec88938\") " pod="openshift-network-operator/iptables-alerter-tzdhp"
Aug 04 18:00:03.177744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-registration-dir\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.177744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfx6r\" (UniqueName: \"kubernetes.io/projected/834b97ef-bbb4-44bd-a065-8408f4523f06-kube-api-access-gfx6r\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.177744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhs2x\" (UniqueName: \"kubernetes.io/projected/7dce6b12-1695-48e9-b13a-0bf64d3107ff-kube-api-access-jhs2x\") pod \"node-ca-7f2bk\" (UID: \"7dce6b12-1695-48e9-b13a-0bf64d3107ff\") " pod="openshift-image-registry/node-ca-7f2bk"
Aug 04 18:00:03.177744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177720    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-cnibin\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.177744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177726    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2959820d-b173-49ec-875d-9dfa00f47f50-os-release\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177760    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2959820d-b173-49ec-875d-9dfa00f47f50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/883bdd22-2b82-4cd5-bc0b-28555ec88938-host-slash\") pod \"iptables-alerter-tzdhp\" (UID: \"883bdd22-2b82-4cd5-bc0b-28555ec88938\") " pod="openshift-network-operator/iptables-alerter-tzdhp"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177800    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-registration-dir\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177819    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54a81028-407e-4ddd-b7bd-9698d6e35f2e-agent-certs\") pod \"konnectivity-agent-wjjnh\" (UID: \"54a81028-407e-4ddd-b7bd-9698d6e35f2e\") " pod="kube-system/konnectivity-agent-wjjnh"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htp46\" (UniqueName: \"kubernetes.io/projected/460132b7-d6bd-44f3-b852-1de1afa84bf8-kube-api-access-htp46\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-os-release\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177903    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-run\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-kubelet\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177948    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-multus-conf-dir\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.177984    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-log-socket\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178011    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-env-overrides\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-ovn-node-metrics-cert\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178069    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret\") pod \"global-pull-secret-syncer-t46pw\" (UID: \"6cbc3f49-e03c-425d-9da2-71eb15915d42\") " pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178096    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-multus-socket-dir-parent\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178118    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09c8e863-111d-467c-9c07-d0d6758b2e20-multus-daemon-config\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/703eadd7-667a-4879-aa4e-2ccedb5b7c8a-tmp-dir\") pod \"node-resolver-v69qf\" (UID: \"703eadd7-667a-4879-aa4e-2ccedb5b7c8a\") " pod="openshift-dns/node-resolver-v69qf"
Aug 04 18:00:03.178382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178188    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-ovnkube-config\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-multus-cni-dir\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178245    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-run-multus-certs\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/883bdd22-2b82-4cd5-bc0b-28555ec88938-iptables-alerter-script\") pod \"iptables-alerter-tzdhp\" (UID: \"883bdd22-2b82-4cd5-bc0b-28555ec88938\") " pod="openshift-network-operator/iptables-alerter-tzdhp"
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178300    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-kubernetes\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178322    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-systemd\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-run-openvswitch\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178427    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2959820d-b173-49ec-875d-9dfa00f47f50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178434    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-ovnkube-script-lib\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-var-lib-cni-bin\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-system-cni-dir\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.178487    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178495    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2959820d-b173-49ec-875d-9dfa00f47f50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7cv6\" (UniqueName: \"kubernetes.io/projected/2959820d-b173-49ec-875d-9dfa00f47f50-kube-api-access-c7cv6\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178548    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-sysctl-d\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.178579    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret podName:6cbc3f49-e03c-425d-9da2-71eb15915d42 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:03.678544402 +0000 UTC m=+2.969158572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret") pod "global-pull-secret-syncer-t46pw" (UID: "6cbc3f49-e03c-425d-9da2-71eb15915d42") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178591    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-multus-socket-dir-parent\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.179150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178609    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dce6b12-1695-48e9-b13a-0bf64d3107ff-serviceca\") pod \"node-ca-7f2bk\" (UID: \"7dce6b12-1695-48e9-b13a-0bf64d3107ff\") " pod="openshift-image-registry/node-ca-7f2bk"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178628    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-var-lib-cni-bin\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178641    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-os-release\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178824    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/703eadd7-667a-4879-aa4e-2ccedb5b7c8a-tmp-dir\") pod \"node-resolver-v69qf\" (UID: \"703eadd7-667a-4879-aa4e-2ccedb5b7c8a\") " pod="openshift-dns/node-resolver-v69qf"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.178875    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-run-multus-certs\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179000    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-multus-cni-dir\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179041    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2959820d-b173-49ec-875d-9dfa00f47f50-cni-binary-copy\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179092    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-var-lib-kubelet\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179089    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2959820d-b173-49ec-875d-9dfa00f47f50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-cni-netd\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179157    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6cbc3f49-e03c-425d-9da2-71eb15915d42-kubelet-config\") pod \"global-pull-secret-syncer-t46pw\" (UID: \"6cbc3f49-e03c-425d-9da2-71eb15915d42\") " pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-var-lib-kubelet\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179220    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2959820d-b173-49ec-875d-9dfa00f47f50-cnibin\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179276    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179285    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6cbc3f49-e03c-425d-9da2-71eb15915d42-kubelet-config\") pod \"global-pull-secret-syncer-t46pw\" (UID: \"6cbc3f49-e03c-425d-9da2-71eb15915d42\") " pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2959820d-b173-49ec-875d-9dfa00f47f50-cnibin\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179303    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-cni-bin\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.180064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179320    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-var-lib-kubelet\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g2x7\" (UniqueName: \"kubernetes.io/projected/09c8e863-111d-467c-9c07-d0d6758b2e20-kube-api-access-9g2x7\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179398    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t95p\" (UniqueName: \"kubernetes.io/projected/703eadd7-667a-4879-aa4e-2ccedb5b7c8a-kube-api-access-5t95p\") pod \"node-resolver-v69qf\" (UID: \"703eadd7-667a-4879-aa4e-2ccedb5b7c8a\") " pod="openshift-dns/node-resolver-v69qf"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179413    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/883bdd22-2b82-4cd5-bc0b-28555ec88938-iptables-alerter-script\") pod \"iptables-alerter-tzdhp\" (UID: \"883bdd22-2b82-4cd5-bc0b-28555ec88938\") " pod="openshift-network-operator/iptables-alerter-tzdhp"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179424    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnpvd\" (UniqueName: \"kubernetes.io/projected/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-kube-api-access-cnpvd\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-run-netns\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179492    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54a81028-407e-4ddd-b7bd-9698d6e35f2e-konnectivity-ca\") pod \"konnectivity-agent-wjjnh\" (UID: \"54a81028-407e-4ddd-b7bd-9698d6e35f2e\") " pod="kube-system/konnectivity-agent-wjjnh"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-systemd-units\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179539    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-run-netns\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179608    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6cbc3f49-e03c-425d-9da2-71eb15915d42-dbus\") pod \"global-pull-secret-syncer-t46pw\" (UID: \"6cbc3f49-e03c-425d-9da2-71eb15915d42\") " pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179655    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-etc-kubernetes\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179675    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6cbc3f49-e03c-425d-9da2-71eb15915d42-dbus\") pod \"global-pull-secret-syncer-t46pw\" (UID: \"6cbc3f49-e03c-425d-9da2-71eb15915d42\") " pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179685    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpj44\" (UniqueName: \"kubernetes.io/projected/47e059c0-7685-4231-86b6-5dbea4f4d5d5-kube-api-access-kpj44\") pod \"network-metrics-daemon-647mg\" (UID: \"47e059c0-7685-4231-86b6-5dbea4f4d5d5\") " pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179710    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-etc-kubernetes\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-sysconfig\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-sys-fs\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2959820d-b173-49ec-875d-9dfa00f47f50-system-cni-dir\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.180873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179840    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2959820d-b173-49ec-875d-9dfa00f47f50-system-cni-dir\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-sysctl-conf\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179917    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-sys-fs\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-run-k8s-cni-cncf-io\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-hostroot\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179983    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-device-dir\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.179993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-run-k8s-cni-cncf-io\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180006    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-etc-selinux\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180031    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2959820d-b173-49ec-875d-9dfa00f47f50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180053    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-hostroot\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180056    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27k4v\" (UniqueName: \"kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v\") pod \"network-check-target-bd4m4\" (UID: \"dbbadf1e-6e33-48b0-8d9c-437c756cb1eb\") " pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-etc-selinux\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180170    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-modprobe-d\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180195    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-node-log\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180203    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2959820d-b173-49ec-875d-9dfa00f47f50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09c8e863-111d-467c-9c07-d0d6758b2e20-cni-binary-copy\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180240    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-device-dir\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.181518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/703eadd7-667a-4879-aa4e-2ccedb5b7c8a-hosts-file\") pod \"node-resolver-v69qf\" (UID: \"703eadd7-667a-4879-aa4e-2ccedb5b7c8a\") " pod="openshift-dns/node-resolver-v69qf"
Aug 04 18:00:03.182091 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180298    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-socket-dir\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.182091 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180335    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-sys\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.182091 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-lib-modules\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.182091 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180404    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/703eadd7-667a-4879-aa4e-2ccedb5b7c8a-hosts-file\") pod \"node-resolver-v69qf\" (UID: \"703eadd7-667a-4879-aa4e-2ccedb5b7c8a\") " pod="openshift-dns/node-resolver-v69qf"
Aug 04 18:00:03.182091 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-var-lib-cni-multus\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.182091 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.182091 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180521    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs\") pod \"network-metrics-daemon-647mg\" (UID: \"47e059c0-7685-4231-86b6-5dbea4f4d5d5\") " pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:03.182091 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180540    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-socket-dir\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.182091 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180570    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09c8e863-111d-467c-9c07-d0d6758b2e20-host-var-lib-cni-multus\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.182091 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180586    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/834b97ef-bbb4-44bd-a065-8408f4523f06-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.182091 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-run-netns\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.182091 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.180680    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09c8e863-111d-467c-9c07-d0d6758b2e20-cni-binary-copy\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.191523 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.191502    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Aug 04 18:00:03.195213 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.195180    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t95p\" (UniqueName: \"kubernetes.io/projected/703eadd7-667a-4879-aa4e-2ccedb5b7c8a-kube-api-access-5t95p\") pod \"node-resolver-v69qf\" (UID: \"703eadd7-667a-4879-aa4e-2ccedb5b7c8a\") " pod="openshift-dns/node-resolver-v69qf"
Aug 04 18:00:03.195323 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.195186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c68d\" (UniqueName: \"kubernetes.io/projected/883bdd22-2b82-4cd5-bc0b-28555ec88938-kube-api-access-9c68d\") pod \"iptables-alerter-tzdhp\" (UID: \"883bdd22-2b82-4cd5-bc0b-28555ec88938\") " pod="openshift-network-operator/iptables-alerter-tzdhp"
Aug 04 18:00:03.195323 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.195186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7cv6\" (UniqueName: \"kubernetes.io/projected/2959820d-b173-49ec-875d-9dfa00f47f50-kube-api-access-c7cv6\") pod \"multus-additional-cni-plugins-hmzpt\" (UID: \"2959820d-b173-49ec-875d-9dfa00f47f50\") " pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.195323 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.195277    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g2x7\" (UniqueName: \"kubernetes.io/projected/09c8e863-111d-467c-9c07-d0d6758b2e20-kube-api-access-9g2x7\") pod \"multus-82njr\" (UID: \"09c8e863-111d-467c-9c07-d0d6758b2e20\") " pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.195323 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.195193    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfx6r\" (UniqueName: \"kubernetes.io/projected/834b97ef-bbb4-44bd-a065-8408f4523f06-kube-api-access-gfx6r\") pod \"aws-ebs-csi-driver-node-ddvjw\" (UID: \"834b97ef-bbb4-44bd-a065-8408f4523f06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.195323 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.195273    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhs2x\" (UniqueName: \"kubernetes.io/projected/7dce6b12-1695-48e9-b13a-0bf64d3107ff-kube-api-access-jhs2x\") pod \"node-ca-7f2bk\" (UID: \"7dce6b12-1695-48e9-b13a-0bf64d3107ff\") " pod="openshift-image-registry/node-ca-7f2bk"
Aug 04 18:00:03.249469 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:03.249436    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b78a122ed0f6c6c547f0087e492e65b.slice/crio-217539d31cb331d73e507872c7bbc03dba0a9c70cd0a5f6895862a74c2a93d6c WatchSource:0}: Error finding container 217539d31cb331d73e507872c7bbc03dba0a9c70cd0a5f6895862a74c2a93d6c: Status 404 returned error can't find the container with id 217539d31cb331d73e507872c7bbc03dba0a9c70cd0a5f6895862a74c2a93d6c
Aug 04 18:00:03.254328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.254308    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:00:03.281339 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281305    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-ovnkube-script-lib\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.281511 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281383    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-sysctl-d\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.281511 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281407    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-var-lib-kubelet\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.281511 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281428    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-cni-netd\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.281511 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.281511 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281473    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-cni-bin\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.281511 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281498    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnpvd\" (UniqueName: \"kubernetes.io/projected/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-kube-api-access-cnpvd\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.281804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54a81028-407e-4ddd-b7bd-9698d6e35f2e-konnectivity-ca\") pod \"konnectivity-agent-wjjnh\" (UID: \"54a81028-407e-4ddd-b7bd-9698d6e35f2e\") " pod="kube-system/konnectivity-agent-wjjnh"
Aug 04 18:00:03.281804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281543    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-systemd-units\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.281804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281557    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.281804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281567    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-sysctl-d\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.281804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281571    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpj44\" (UniqueName: \"kubernetes.io/projected/47e059c0-7685-4231-86b6-5dbea4f4d5d5-kube-api-access-kpj44\") pod \"network-metrics-daemon-647mg\" (UID: \"47e059c0-7685-4231-86b6-5dbea4f4d5d5\") " pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:03.281804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281648    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-sysconfig\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.281804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281704    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-sysctl-conf\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.281804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281741    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-cni-netd\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.281804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27k4v\" (UniqueName: \"kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v\") pod \"network-check-target-bd4m4\" (UID: \"dbbadf1e-6e33-48b0-8d9c-437c756cb1eb\") " pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:03.281804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281787    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-modprobe-d\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.281804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281804    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-systemd-units\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281815    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-node-log\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281705    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-cni-bin\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-sys\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281847    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-var-lib-kubelet\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281866    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-node-log\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281883    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-sysconfig\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281913    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-modprobe-d\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-lib-modules\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281956    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs\") pod \"network-metrics-daemon-647mg\" (UID: \"47e059c0-7685-4231-86b6-5dbea4f4d5d5\") " pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281957    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-sysctl-conf\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281985    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-ovnkube-script-lib\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.281997    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-sys\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-lib-modules\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-run-netns\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282073    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-run-systemd\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282080    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-run-netns\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.282112    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:03.282270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282117    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-var-lib-openvswitch\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-var-lib-openvswitch\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-run-ovn\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282181    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-run-systemd\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.282233    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs podName:47e059c0-7685-4231-86b6-5dbea4f4d5d5 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:03.782218892 +0000 UTC m=+3.072833046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs") pod "network-metrics-daemon-647mg" (UID: "47e059c0-7685-4231-86b6-5dbea4f4d5d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282242    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-run-ovn\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282315    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-tuned\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/460132b7-d6bd-44f3-b852-1de1afa84bf8-tmp\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-etc-openvswitch\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282429    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-etc-openvswitch\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-slash\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282476    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282502    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-slash\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-host\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282563    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54a81028-407e-4ddd-b7bd-9698d6e35f2e-agent-certs\") pod \"konnectivity-agent-wjjnh\" (UID: \"54a81028-407e-4ddd-b7bd-9698d6e35f2e\") " pod="kube-system/konnectivity-agent-wjjnh"
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282588    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htp46\" (UniqueName: \"kubernetes.io/projected/460132b7-d6bd-44f3-b852-1de1afa84bf8-kube-api-access-htp46\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.283112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-run\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-kubelet\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-log-socket\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-env-overrides\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282710    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-ovn-node-metrics-cert\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282764    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-host\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282769    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-ovnkube-config\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282784    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54a81028-407e-4ddd-b7bd-9698d6e35f2e-konnectivity-ca\") pod \"konnectivity-agent-wjjnh\" (UID: \"54a81028-407e-4ddd-b7bd-9698d6e35f2e\") " pod="kube-system/konnectivity-agent-wjjnh"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282870    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-host-kubelet\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282927    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-log-socket\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282820    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-kubernetes\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282973    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-run\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.282981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-systemd\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.283006    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-run-openvswitch\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.283043    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-kubernetes\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.283075    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-run-openvswitch\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.283074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-systemd\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.283312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-env-overrides\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.283805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.283434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-ovnkube-config\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.284590 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.284570    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/460132b7-d6bd-44f3-b852-1de1afa84bf8-tmp\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.284826 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.284811    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/460132b7-d6bd-44f3-b852-1de1afa84bf8-etc-tuned\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.285396 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.285381    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54a81028-407e-4ddd-b7bd-9698d6e35f2e-agent-certs\") pod \"konnectivity-agent-wjjnh\" (UID: \"54a81028-407e-4ddd-b7bd-9698d6e35f2e\") " pod="kube-system/konnectivity-agent-wjjnh"
Aug 04 18:00:03.285555 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.285541    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-ovn-node-metrics-cert\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.290996 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.290978    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:03.291073 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.291001    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:03.291073 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.291016    2566 projected.go:196] Error preparing data for projected volume kube-api-access-27k4v for pod openshift-network-diagnostics/network-check-target-bd4m4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:03.291151 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.291080    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v podName:dbbadf1e-6e33-48b0-8d9c-437c756cb1eb nodeName:}" failed. No retries permitted until 2026-08-04 18:00:03.791062154 +0000 UTC m=+3.081676307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-27k4v" (UniqueName: "kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v") pod "network-check-target-bd4m4" (UID: "dbbadf1e-6e33-48b0-8d9c-437c756cb1eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:03.292898 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.292870    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpj44\" (UniqueName: \"kubernetes.io/projected/47e059c0-7685-4231-86b6-5dbea4f4d5d5-kube-api-access-kpj44\") pod \"network-metrics-daemon-647mg\" (UID: \"47e059c0-7685-4231-86b6-5dbea4f4d5d5\") " pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:03.293121 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.293103    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htp46\" (UniqueName: \"kubernetes.io/projected/460132b7-d6bd-44f3-b852-1de1afa84bf8-kube-api-access-htp46\") pod \"tuned-lfd9z\" (UID: \"460132b7-d6bd-44f3-b852-1de1afa84bf8\") " pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.294019 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.294002    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnpvd\" (UniqueName: \"kubernetes.io/projected/e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e-kube-api-access-cnpvd\") pod \"ovnkube-node-kfzm2\" (UID: \"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.397055 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.396959    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v69qf"
Aug 04 18:00:03.402332 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.402308    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw"
Aug 04 18:00:03.408368 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:03.408327    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703eadd7_667a_4879_aa4e_2ccedb5b7c8a.slice/crio-a31a7d35219933ca2484e1e21da68e68d3c816f4246c85554afe3ab2be90e224 WatchSource:0}: Error finding container a31a7d35219933ca2484e1e21da68e68d3c816f4246c85554afe3ab2be90e224: Status 404 returned error can't find the container with id a31a7d35219933ca2484e1e21da68e68d3c816f4246c85554afe3ab2be90e224
Aug 04 18:00:03.413132 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:03.413112    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834b97ef_bbb4_44bd_a065_8408f4523f06.slice/crio-acd5c8df538c3114619281b925a92f674763df759db99ea4c088a9ccbea4b82d WatchSource:0}: Error finding container acd5c8df538c3114619281b925a92f674763df759db99ea4c088a9ccbea4b82d: Status 404 returned error can't find the container with id acd5c8df538c3114619281b925a92f674763df759db99ea4c088a9ccbea4b82d
Aug 04 18:00:03.437081 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.437045    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7f2bk"
Aug 04 18:00:03.446023 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:03.445996    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dce6b12_1695_48e9_b13a_0bf64d3107ff.slice/crio-02436d93b970e1e4d5eab235d25157f2751983aa8f6cfc58a881ca9e4ba53b0e WatchSource:0}: Error finding container 02436d93b970e1e4d5eab235d25157f2751983aa8f6cfc58a881ca9e4ba53b0e: Status 404 returned error can't find the container with id 02436d93b970e1e4d5eab235d25157f2751983aa8f6cfc58a881ca9e4ba53b0e
Aug 04 18:00:03.454362 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.454331    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-82njr"
Aug 04 18:00:03.472187 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:03.472158    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7cee5e53dc47fb7193759f38d87999.slice/crio-30c86ee34a08ba2b3279aa36dbecae15e968296df64ea1898a695be546e05de7 WatchSource:0}: Error finding container 30c86ee34a08ba2b3279aa36dbecae15e968296df64ea1898a695be546e05de7: Status 404 returned error can't find the container with id 30c86ee34a08ba2b3279aa36dbecae15e968296df64ea1898a695be546e05de7
Aug 04 18:00:03.472471 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:03.472446    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c8e863_111d_467c_9c07_d0d6758b2e20.slice/crio-3a405773e6946f795a2155e8a088094aa57c96127892cfbdd021fc863bbc060d WatchSource:0}: Error finding container 3a405773e6946f795a2155e8a088094aa57c96127892cfbdd021fc863bbc060d: Status 404 returned error can't find the container with id 3a405773e6946f795a2155e8a088094aa57c96127892cfbdd021fc863bbc060d
Aug 04 18:00:03.473177 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.473162    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hmzpt"
Aug 04 18:00:03.482022 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.482002    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tzdhp"
Aug 04 18:00:03.482302 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:03.482280    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2959820d_b173_49ec_875d_9dfa00f47f50.slice/crio-1d6fdc1b6a5acd62c4077b069cd90f202ad02038568f227852cba70118fd5b33 WatchSource:0}: Error finding container 1d6fdc1b6a5acd62c4077b069cd90f202ad02038568f227852cba70118fd5b33: Status 404 returned error can't find the container with id 1d6fdc1b6a5acd62c4077b069cd90f202ad02038568f227852cba70118fd5b33
Aug 04 18:00:03.490441 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:03.490421    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883bdd22_2b82_4cd5_bc0b_28555ec88938.slice/crio-20ac99756aa04273a22a74bbe12e774a0ec61404e2913f0085776b69a55cc5d8 WatchSource:0}: Error finding container 20ac99756aa04273a22a74bbe12e774a0ec61404e2913f0085776b69a55cc5d8: Status 404 returned error can't find the container with id 20ac99756aa04273a22a74bbe12e774a0ec61404e2913f0085776b69a55cc5d8
Aug 04 18:00:03.495593 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.495576    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:03.496721 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.496705    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:03.504779 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:03.504759    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40e6de3_aae2_44dc_aa01_ed1de3d1ce5e.slice/crio-e3bb812108c51b8b322efa50337017c636f1923efd24b0ffa55f1bcf5b52e80e WatchSource:0}: Error finding container e3bb812108c51b8b322efa50337017c636f1923efd24b0ffa55f1bcf5b52e80e: Status 404 returned error can't find the container with id e3bb812108c51b8b322efa50337017c636f1923efd24b0ffa55f1bcf5b52e80e
Aug 04 18:00:03.514798 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.514780    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wjjnh"
Aug 04 18:00:03.519617 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.519601    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lfd9z"
Aug 04 18:00:03.522475 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:03.522454    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a81028_407e_4ddd_b7bd_9698d6e35f2e.slice/crio-eec11058dd6a8aad39c8717122562ef0098bbaaaba2f8fb07e8d26cf32531d00 WatchSource:0}: Error finding container eec11058dd6a8aad39c8717122562ef0098bbaaaba2f8fb07e8d26cf32531d00: Status 404 returned error can't find the container with id eec11058dd6a8aad39c8717122562ef0098bbaaaba2f8fb07e8d26cf32531d00
Aug 04 18:00:03.527569 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:03.527549    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460132b7_d6bd_44f3_b852_1de1afa84bf8.slice/crio-37439146dc87b560be907e8e4500b03c88fdd5900944c1866b2d15a9a4756833 WatchSource:0}: Error finding container 37439146dc87b560be907e8e4500b03c88fdd5900944c1866b2d15a9a4756833: Status 404 returned error can't find the container with id 37439146dc87b560be907e8e4500b03c88fdd5900944c1866b2d15a9a4756833
Aug 04 18:00:03.684521 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.684422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret\") pod \"global-pull-secret-syncer-t46pw\" (UID: \"6cbc3f49-e03c-425d-9da2-71eb15915d42\") " pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:03.684670 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.684558    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:03.684670 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.684626    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret podName:6cbc3f49-e03c-425d-9da2-71eb15915d42 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:04.684607851 +0000 UTC m=+3.975222002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret") pod "global-pull-secret-syncer-t46pw" (UID: "6cbc3f49-e03c-425d-9da2-71eb15915d42") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:03.785204 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.785172    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs\") pod \"network-metrics-daemon-647mg\" (UID: \"47e059c0-7685-4231-86b6-5dbea4f4d5d5\") " pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:03.785398 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.785289    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:03.785398 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.785344    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs podName:47e059c0-7685-4231-86b6-5dbea4f4d5d5 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:04.785331477 +0000 UTC m=+4.075945626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs") pod "network-metrics-daemon-647mg" (UID: "47e059c0-7685-4231-86b6-5dbea4f4d5d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:03.886507 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:03.886471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27k4v\" (UniqueName: \"kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v\") pod \"network-check-target-bd4m4\" (UID: \"dbbadf1e-6e33-48b0-8d9c-437c756cb1eb\") " pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:03.886673 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.886631    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:03.886673 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.886659    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:03.886673 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.886669    2566 projected.go:196] Error preparing data for projected volume kube-api-access-27k4v for pod openshift-network-diagnostics/network-check-target-bd4m4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:03.886782 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:03.886718    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v podName:dbbadf1e-6e33-48b0-8d9c-437c756cb1eb nodeName:}" failed. No retries permitted until 2026-08-04 18:00:04.886704857 +0000 UTC m=+4.177319006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-27k4v" (UniqueName: "kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v") pod "network-check-target-bd4m4" (UID: "dbbadf1e-6e33-48b0-8d9c-437c756cb1eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:04.140375 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.140262    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:55:03 +0000 UTC" deadline="2028-01-23 12:11:34.761880826 +0000 UTC"
Aug 04 18:00:04.140375 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.140295    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12882h11m30.621589505s"
Aug 04 18:00:04.188726 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.188698    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:04.188886 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:04.188796    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:04.194347 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.194305    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2" event={"ID":"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e","Type":"ContainerStarted","Data":"e3bb812108c51b8b322efa50337017c636f1923efd24b0ffa55f1bcf5b52e80e"}
Aug 04 18:00:04.195943 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.195924    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-82njr" event={"ID":"09c8e863-111d-467c-9c07-d0d6758b2e20","Type":"ContainerStarted","Data":"3a405773e6946f795a2155e8a088094aa57c96127892cfbdd021fc863bbc060d"}
Aug 04 18:00:04.196870 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.196841    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7f2bk" event={"ID":"7dce6b12-1695-48e9-b13a-0bf64d3107ff","Type":"ContainerStarted","Data":"02436d93b970e1e4d5eab235d25157f2751983aa8f6cfc58a881ca9e4ba53b0e"}
Aug 04 18:00:04.197796 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.197778    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v69qf" event={"ID":"703eadd7-667a-4879-aa4e-2ccedb5b7c8a","Type":"ContainerStarted","Data":"a31a7d35219933ca2484e1e21da68e68d3c816f4246c85554afe3ab2be90e224"}
Aug 04 18:00:04.198813 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.198796    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lfd9z" event={"ID":"460132b7-d6bd-44f3-b852-1de1afa84bf8","Type":"ContainerStarted","Data":"37439146dc87b560be907e8e4500b03c88fdd5900944c1866b2d15a9a4756833"}
Aug 04 18:00:04.199801 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.199784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wjjnh" event={"ID":"54a81028-407e-4ddd-b7bd-9698d6e35f2e","Type":"ContainerStarted","Data":"eec11058dd6a8aad39c8717122562ef0098bbaaaba2f8fb07e8d26cf32531d00"}
Aug 04 18:00:04.200810 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.200792    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tzdhp" event={"ID":"883bdd22-2b82-4cd5-bc0b-28555ec88938","Type":"ContainerStarted","Data":"20ac99756aa04273a22a74bbe12e774a0ec61404e2913f0085776b69a55cc5d8"}
Aug 04 18:00:04.201693 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.201674    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hmzpt" event={"ID":"2959820d-b173-49ec-875d-9dfa00f47f50","Type":"ContainerStarted","Data":"1d6fdc1b6a5acd62c4077b069cd90f202ad02038568f227852cba70118fd5b33"}
Aug 04 18:00:04.202699 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.202682    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal" event={"ID":"3d7cee5e53dc47fb7193759f38d87999","Type":"ContainerStarted","Data":"30c86ee34a08ba2b3279aa36dbecae15e968296df64ea1898a695be546e05de7"}
Aug 04 18:00:04.203690 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.203664    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw" event={"ID":"834b97ef-bbb4-44bd-a065-8408f4523f06","Type":"ContainerStarted","Data":"acd5c8df538c3114619281b925a92f674763df759db99ea4c088a9ccbea4b82d"}
Aug 04 18:00:04.204586 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.204570    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-242.ec2.internal" event={"ID":"4b78a122ed0f6c6c547f0087e492e65b","Type":"ContainerStarted","Data":"217539d31cb331d73e507872c7bbc03dba0a9c70cd0a5f6895862a74c2a93d6c"}
Aug 04 18:00:04.692140 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.691595    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret\") pod \"global-pull-secret-syncer-t46pw\" (UID: \"6cbc3f49-e03c-425d-9da2-71eb15915d42\") " pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:04.692140 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:04.691732    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:04.692140 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:04.691808    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret podName:6cbc3f49-e03c-425d-9da2-71eb15915d42 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:06.691788593 +0000 UTC m=+5.982402756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret") pod "global-pull-secret-syncer-t46pw" (UID: "6cbc3f49-e03c-425d-9da2-71eb15915d42") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:04.792527 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.792489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs\") pod \"network-metrics-daemon-647mg\" (UID: \"47e059c0-7685-4231-86b6-5dbea4f4d5d5\") " pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:04.792707 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:04.792633    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:04.792707 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:04.792697    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs podName:47e059c0-7685-4231-86b6-5dbea4f4d5d5 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:06.792678297 +0000 UTC m=+6.083292459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs") pod "network-metrics-daemon-647mg" (UID: "47e059c0-7685-4231-86b6-5dbea4f4d5d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:04.893294 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:04.893239    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27k4v\" (UniqueName: \"kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v\") pod \"network-check-target-bd4m4\" (UID: \"dbbadf1e-6e33-48b0-8d9c-437c756cb1eb\") " pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:04.893500 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:04.893434    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:04.893500 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:04.893458    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:04.893500 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:04.893473    2566 projected.go:196] Error preparing data for projected volume kube-api-access-27k4v for pod openshift-network-diagnostics/network-check-target-bd4m4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:04.893666 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:04.893542    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v podName:dbbadf1e-6e33-48b0-8d9c-437c756cb1eb nodeName:}" failed. No retries permitted until 2026-08-04 18:00:06.893523247 +0000 UTC m=+6.184137418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-27k4v" (UniqueName: "kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v") pod "network-check-target-bd4m4" (UID: "dbbadf1e-6e33-48b0-8d9c-437c756cb1eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:05.141262 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:05.141188    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:55:03 +0000 UTC" deadline="2028-02-19 12:12:55.927670429 +0000 UTC"
Aug 04 18:00:05.141262 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:05.141261    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13530h12m50.786413872s"
Aug 04 18:00:05.189517 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:05.189486    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:05.189681 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:05.189636    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:05.190423 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:05.190172    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:05.190423 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:05.190293    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:06.189221 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:06.189186    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:06.189713 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:06.189395    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:06.710365 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:06.710313    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret\") pod \"global-pull-secret-syncer-t46pw\" (UID: \"6cbc3f49-e03c-425d-9da2-71eb15915d42\") " pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:06.710556 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:06.710472    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:06.710616 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:06.710587    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret podName:6cbc3f49-e03c-425d-9da2-71eb15915d42 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:10.710566533 +0000 UTC m=+10.001180686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret") pod "global-pull-secret-syncer-t46pw" (UID: "6cbc3f49-e03c-425d-9da2-71eb15915d42") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:06.811720 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:06.811053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs\") pod \"network-metrics-daemon-647mg\" (UID: \"47e059c0-7685-4231-86b6-5dbea4f4d5d5\") " pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:06.811720 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:06.811249    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:06.811720 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:06.811320    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs podName:47e059c0-7685-4231-86b6-5dbea4f4d5d5 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:10.811298395 +0000 UTC m=+10.101912544 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs") pod "network-metrics-daemon-647mg" (UID: "47e059c0-7685-4231-86b6-5dbea4f4d5d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:06.912023 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:06.911982    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27k4v\" (UniqueName: \"kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v\") pod \"network-check-target-bd4m4\" (UID: \"dbbadf1e-6e33-48b0-8d9c-437c756cb1eb\") " pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:06.912212 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:06.912172    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:06.912212 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:06.912193    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:06.912212 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:06.912205    2566 projected.go:196] Error preparing data for projected volume kube-api-access-27k4v for pod openshift-network-diagnostics/network-check-target-bd4m4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:06.912469 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:06.912264    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v podName:dbbadf1e-6e33-48b0-8d9c-437c756cb1eb nodeName:}" failed. No retries permitted until 2026-08-04 18:00:10.912245868 +0000 UTC m=+10.202860020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-27k4v" (UniqueName: "kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v") pod "network-check-target-bd4m4" (UID: "dbbadf1e-6e33-48b0-8d9c-437c756cb1eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:07.192064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:07.191105    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:07.192064 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:07.191241    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:07.192064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:07.191622    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:07.192064 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:07.191712    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:08.189135 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:08.189096    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:08.189314 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:08.189269    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:09.189771 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:09.189249    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:09.189771 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:09.189409    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:09.190536 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:09.190365    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:09.190536 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:09.190481    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:10.189108 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:10.188842    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:10.189108 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:10.188985    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:10.743540 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:10.743491    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret\") pod \"global-pull-secret-syncer-t46pw\" (UID: \"6cbc3f49-e03c-425d-9da2-71eb15915d42\") " pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:10.744005 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:10.743656    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:10.744005 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:10.743721    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret podName:6cbc3f49-e03c-425d-9da2-71eb15915d42 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:18.743702552 +0000 UTC m=+18.034316702 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret") pod "global-pull-secret-syncer-t46pw" (UID: "6cbc3f49-e03c-425d-9da2-71eb15915d42") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:10.844715 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:10.844677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs\") pod \"network-metrics-daemon-647mg\" (UID: \"47e059c0-7685-4231-86b6-5dbea4f4d5d5\") " pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:10.844879 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:10.844835    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:10.844954 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:10.844885    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs podName:47e059c0-7685-4231-86b6-5dbea4f4d5d5 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:18.844871715 +0000 UTC m=+18.135485864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs") pod "network-metrics-daemon-647mg" (UID: "47e059c0-7685-4231-86b6-5dbea4f4d5d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:10.945275 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:10.945221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27k4v\" (UniqueName: \"kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v\") pod \"network-check-target-bd4m4\" (UID: \"dbbadf1e-6e33-48b0-8d9c-437c756cb1eb\") " pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:10.945484 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:10.945462    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:10.945484 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:10.945483    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:10.945636 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:10.945497    2566 projected.go:196] Error preparing data for projected volume kube-api-access-27k4v for pod openshift-network-diagnostics/network-check-target-bd4m4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:10.945636 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:10.945572    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v podName:dbbadf1e-6e33-48b0-8d9c-437c756cb1eb nodeName:}" failed. No retries permitted until 2026-08-04 18:00:18.945555631 +0000 UTC m=+18.236169782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-27k4v" (UniqueName: "kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v") pod "network-check-target-bd4m4" (UID: "dbbadf1e-6e33-48b0-8d9c-437c756cb1eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:11.190597 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:11.190028    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:11.190597 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:11.190242    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:11.190597 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:11.190435    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:11.190597 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:11.190515    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:12.190068 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:12.189540    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:12.190068 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:12.189682    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:13.189330 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:13.189294    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:13.189595 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:13.189457    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:13.189940 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:13.189916    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:13.190063 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:13.190040    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:14.188837 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:14.188798    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:14.189286 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:14.188927    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:15.188848 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:15.188810    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:15.189306 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:15.188828    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:15.189306 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:15.188968    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:15.189306 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:15.189048    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:16.188639 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:16.188611    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:16.188817 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:16.188709    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:17.189078 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:17.189043    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:17.189561 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:17.189048    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:17.189561 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:17.189167    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:17.189561 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:17.189275    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:18.188559 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:18.188520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:18.188740 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:18.188639    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:18.811202 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:18.811167    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret\") pod \"global-pull-secret-syncer-t46pw\" (UID: \"6cbc3f49-e03c-425d-9da2-71eb15915d42\") " pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:18.811652 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:18.811279    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:18.811652 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:18.811344    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret podName:6cbc3f49-e03c-425d-9da2-71eb15915d42 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:34.811325886 +0000 UTC m=+34.101940039 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret") pod "global-pull-secret-syncer-t46pw" (UID: "6cbc3f49-e03c-425d-9da2-71eb15915d42") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:18.911692 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:18.911659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs\") pod \"network-metrics-daemon-647mg\" (UID: \"47e059c0-7685-4231-86b6-5dbea4f4d5d5\") " pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:18.911876 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:18.911787    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:18.911876 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:18.911843    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs podName:47e059c0-7685-4231-86b6-5dbea4f4d5d5 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:34.911826522 +0000 UTC m=+34.202440671 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs") pod "network-metrics-daemon-647mg" (UID: "47e059c0-7685-4231-86b6-5dbea4f4d5d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:19.012698 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:19.012662    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27k4v\" (UniqueName: \"kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v\") pod \"network-check-target-bd4m4\" (UID: \"dbbadf1e-6e33-48b0-8d9c-437c756cb1eb\") " pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:19.012861 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:19.012816    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:19.012861 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:19.012835    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:19.012861 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:19.012844    2566 projected.go:196] Error preparing data for projected volume kube-api-access-27k4v for pod openshift-network-diagnostics/network-check-target-bd4m4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:19.012973 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:19.012894    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v podName:dbbadf1e-6e33-48b0-8d9c-437c756cb1eb nodeName:}" failed. No retries permitted until 2026-08-04 18:00:35.012881154 +0000 UTC m=+34.303495303 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-27k4v" (UniqueName: "kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v") pod "network-check-target-bd4m4" (UID: "dbbadf1e-6e33-48b0-8d9c-437c756cb1eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:19.192552 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:19.192477    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:19.192713 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:19.192484    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:19.192713 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:19.192605    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:19.192713 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:19.192670    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:20.189043 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:20.188997    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:20.189574 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:20.189141    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:21.191482 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:21.191455    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:21.191482 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:21.191471    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:21.191927 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:21.191563    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:21.191927 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:21.191706    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:22.189494 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:22.189460    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:22.189697 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:22.189590    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:23.189055 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:23.189011    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:23.189468 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:23.189147    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:23.189468 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:23.189204    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:23.189468 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:23.189294    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:24.189620 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:24.189440    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:24.189976 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:24.189686    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:25.191943 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.191760    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:25.192591 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.191804    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:25.192591 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:25.192026    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:25.192591 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:25.192123    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:25.249580 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.248564    2566 generic.go:356] "Generic (PLEG): container finished" podID="2959820d-b173-49ec-875d-9dfa00f47f50" containerID="f63bcdbe711b78741cea9771f776dcaba25b740d037d52e5f269d374b65c8d5d" exitCode=0
Aug 04 18:00:25.249580 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.248671    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hmzpt" event={"ID":"2959820d-b173-49ec-875d-9dfa00f47f50","Type":"ContainerDied","Data":"f63bcdbe711b78741cea9771f776dcaba25b740d037d52e5f269d374b65c8d5d"}
Aug 04 18:00:25.252067 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.252038    2566 generic.go:356] "Generic (PLEG): container finished" podID="3d7cee5e53dc47fb7193759f38d87999" containerID="6bca8575d5ba760d6f04e84bc5b5df132417b28da84c3529dda73759fff98e8b" exitCode=0
Aug 04 18:00:25.252174 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.252119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal" event={"ID":"3d7cee5e53dc47fb7193759f38d87999","Type":"ContainerDied","Data":"6bca8575d5ba760d6f04e84bc5b5df132417b28da84c3529dda73759fff98e8b"}
Aug 04 18:00:25.253772 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.253751    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw" event={"ID":"834b97ef-bbb4-44bd-a065-8408f4523f06","Type":"ContainerStarted","Data":"59cde0d110703d723b77740a40c887e134714472970a89f1fe8b6f148b153960"}
Aug 04 18:00:25.255139 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.255117    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-242.ec2.internal" event={"ID":"4b78a122ed0f6c6c547f0087e492e65b","Type":"ContainerStarted","Data":"e9e1e2132af682b7890af3c545f31e4c648b44ad6020071c3ae069312b0d6978"}
Aug 04 18:00:25.258239 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.258220    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2" event={"ID":"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e","Type":"ContainerStarted","Data":"132a38afa49969f4f3ea62af7416c5b29cb120bd3f439f53c01647a987ac684d"}
Aug 04 18:00:25.258321 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.258275    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2" event={"ID":"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e","Type":"ContainerStarted","Data":"a40ac2c4b94b4a1e36dba9edec99c10adefd70da1e5022fd3522574930358957"}
Aug 04 18:00:25.258321 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.258318    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2" event={"ID":"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e","Type":"ContainerStarted","Data":"aacf1dfd5e9f3c3253eba2326b219d6685f47744a79e7bdd2af5ee3348b195a6"}
Aug 04 18:00:25.258432 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.258364    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2" event={"ID":"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e","Type":"ContainerStarted","Data":"2962e2425dcd2067b2663c08727ae57efe00f9c6b55831ac865d19bb785cf247"}
Aug 04 18:00:25.258432 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.258409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2" event={"ID":"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e","Type":"ContainerStarted","Data":"f0b06b24a7f774583dd90c48a747ce49df9eb1e7378e8a2fb34cff7dde485bd6"}
Aug 04 18:00:25.259515 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.259497    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-82njr" event={"ID":"09c8e863-111d-467c-9c07-d0d6758b2e20","Type":"ContainerStarted","Data":"12556c7ec57090af3e4036d9504d9446451c33a7b37e719464d3a72b4964b905"}
Aug 04 18:00:25.260672 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.260656    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7f2bk" event={"ID":"7dce6b12-1695-48e9-b13a-0bf64d3107ff","Type":"ContainerStarted","Data":"633f3e04cf5fc8b12a56d0919dd10e0c3054300b38bcd7f7040a46bede604a3d"}
Aug 04 18:00:25.262018 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.261999    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v69qf" event={"ID":"703eadd7-667a-4879-aa4e-2ccedb5b7c8a","Type":"ContainerStarted","Data":"468dd14fd6cd295375842fe865f39ff619ea0aaa3996fde5891bf435d658b91a"}
Aug 04 18:00:25.263237 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.263218    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lfd9z" event={"ID":"460132b7-d6bd-44f3-b852-1de1afa84bf8","Type":"ContainerStarted","Data":"ce35964e44748309d7d9e97ed8ab9f35a48f5341034f9f38546fd6a6b86fc131"}
Aug 04 18:00:25.264453 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.264436    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wjjnh" event={"ID":"54a81028-407e-4ddd-b7bd-9698d6e35f2e","Type":"ContainerStarted","Data":"289a5e00f9c2ca79de2ae077af5049e54aa2721ee2a61686949fc698e267f56c"}
Aug 04 18:00:25.290601 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.290550    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-82njr" podStartSLOduration=3.242182863 podStartE2EDuration="24.290534768s" totalImagesPullingTime="21.048351905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:25.290474606 +0000 UTC m=+24.581088777" watchObservedRunningTime="2026-08-04 18:00:25.290534768 +0000 UTC m=+24.581148939"
Aug 04 18:00:25.308223 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.308180    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wjjnh" podStartSLOduration=3.426731412 podStartE2EDuration="24.308166005s" totalImagesPullingTime="20.881434593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:25.307963025 +0000 UTC m=+24.598577196" watchObservedRunningTime="2026-08-04 18:00:25.308166005 +0000 UTC m=+24.598780175"
Aug 04 18:00:25.324316 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.324262    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lfd9z" podStartSLOduration=3.417597116 podStartE2EDuration="24.324244734s" totalImagesPullingTime="20.906647618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:25.324217497 +0000 UTC m=+24.614831671" watchObservedRunningTime="2026-08-04 18:00:25.324244734 +0000 UTC m=+24.614858928"
Aug 04 18:00:25.337961 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.337915    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-242.ec2.internal" podStartSLOduration=23.337901028 podStartE2EDuration="23.337901028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:25.337534393 +0000 UTC m=+24.628148580" watchObservedRunningTime="2026-08-04 18:00:25.337901028 +0000 UTC m=+24.628515199"
Aug 04 18:00:25.351740 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:25.351698    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v69qf" podStartSLOduration=3.327382852 podStartE2EDuration="24.351683378s" totalImagesPullingTime="21.024300526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:25.351379113 +0000 UTC m=+24.641993315" watchObservedRunningTime="2026-08-04 18:00:25.351683378 +0000 UTC m=+24.642297549"
Aug 04 18:00:26.189154 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:26.188875    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:26.189377 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:26.189182    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:26.269654 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:26.269612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2" event={"ID":"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e","Type":"ContainerStarted","Data":"a9793678cdf725b505d0a4d199e82f1ff51c81572c5b355727f927437368493a"}
Aug 04 18:00:26.271368 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:26.271303    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tzdhp" event={"ID":"883bdd22-2b82-4cd5-bc0b-28555ec88938","Type":"ContainerStarted","Data":"f25c7f7a6b947b54d5121c47ada941994fe8c827f7ed801285581803885fa9a1"}
Aug 04 18:00:26.288001 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:26.287950    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tzdhp" podStartSLOduration=4.34518135 podStartE2EDuration="25.287931912s" totalImagesPullingTime="20.942750562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:26.287748414 +0000 UTC m=+25.578362603" watchObservedRunningTime="2026-08-04 18:00:26.287931912 +0000 UTC m=+25.578546083"
Aug 04 18:00:26.288305 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:26.288275    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7f2bk" podStartSLOduration=4.301060503 podStartE2EDuration="25.288254819s" totalImagesPullingTime="20.987194316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:25.365276527 +0000 UTC m=+24.655890699" watchObservedRunningTime="2026-08-04 18:00:26.288254819 +0000 UTC m=+25.578868991"
Aug 04 18:00:26.967741 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:26.967699    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wjjnh"
Aug 04 18:00:26.968251 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:26.968232    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wjjnh"
Aug 04 18:00:27.045693 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:27.045474    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Aug 04 18:00:27.140591 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:27.140473    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T18:00:27.045492001Z","UUID":"24391d9a-19bb-4cd1-9f73-37f948fb9f63","Handler":null,"Name":"","Endpoint":""}
Aug 04 18:00:27.142245 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:27.142220    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Aug 04 18:00:27.142413 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:27.142263    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Aug 04 18:00:27.188566 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:27.188491    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:27.188725 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:27.188491    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:27.188725 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:27.188627    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:27.188725 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:27.188668    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:27.276091 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:27.276052    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal" event={"ID":"3d7cee5e53dc47fb7193759f38d87999","Type":"ContainerStarted","Data":"fbbd12334c7a8da58b1af3e7efe6f89d7f793a5e355855f4d6d82639324a0b08"}
Aug 04 18:00:27.278193 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:27.278158    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw" event={"ID":"834b97ef-bbb4-44bd-a065-8408f4523f06","Type":"ContainerStarted","Data":"3d93af13b014ee276b3a8518b11349cf35c48a669da7622d8af5398f67049175"}
Aug 04 18:00:27.278516 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:27.278470    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wjjnh"
Aug 04 18:00:27.278953 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:27.278931    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wjjnh"
Aug 04 18:00:27.294267 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:27.294218    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-242.ec2.internal" podStartSLOduration=25.294203021 podStartE2EDuration="25.294203021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:27.293726381 +0000 UTC m=+26.584340571" watchObservedRunningTime="2026-08-04 18:00:27.294203021 +0000 UTC m=+26.584817193"
Aug 04 18:00:28.188485 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:28.188455    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:28.188731 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:28.188577    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:28.283998 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:28.283961    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2" event={"ID":"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e","Type":"ContainerStarted","Data":"eb1cad8451e09616c25d6a9d359481f701ba2a4b99302d9d8233cdc595e90544"}
Aug 04 18:00:29.188526 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:29.188493    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:29.188526 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:29.188520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:29.189000 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:29.188935    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:29.189115 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:29.189087    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:30.189431 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:30.189234    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:30.189956 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:30.189508    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:30.290404 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:30.290346    2566 generic.go:356] "Generic (PLEG): container finished" podID="2959820d-b173-49ec-875d-9dfa00f47f50" containerID="30b7d1840d89a662ee37da4b5f208aa006541189db18f42c5f4db25d397a2aca" exitCode=0
Aug 04 18:00:30.290575 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:30.290455    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hmzpt" event={"ID":"2959820d-b173-49ec-875d-9dfa00f47f50","Type":"ContainerDied","Data":"30b7d1840d89a662ee37da4b5f208aa006541189db18f42c5f4db25d397a2aca"}
Aug 04 18:00:30.292718 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:30.292677    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw" event={"ID":"834b97ef-bbb4-44bd-a065-8408f4523f06","Type":"ContainerStarted","Data":"1eefb71efedc00dab4d843f0b3904a6bab9ff3872daf0e0ff7ad8c9f90369e2b"}
Aug 04 18:00:30.296364 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:30.296325    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2" event={"ID":"e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e","Type":"ContainerStarted","Data":"aa24d6393a130ec5439217dd08da21d014fcf404572d2136713a392955c9be2b"}
Aug 04 18:00:30.296552 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:30.296539    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:30.296611 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:30.296599    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:30.311746 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:30.311720    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:30.331174 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:30.331134    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ddvjw" podStartSLOduration=3.873078597 podStartE2EDuration="29.331121052s" totalImagesPullingTime="25.458042455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:30.33105877 +0000 UTC m=+29.621672952" watchObservedRunningTime="2026-08-04 18:00:30.331121052 +0000 UTC m=+29.621735223"
Aug 04 18:00:30.356331 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:30.356287    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2" podStartSLOduration=8.386646235 podStartE2EDuration="29.356272815s" totalImagesPullingTime="20.96962658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:30.355919237 +0000 UTC m=+29.646533418" watchObservedRunningTime="2026-08-04 18:00:30.356272815 +0000 UTC m=+29.646886965"
Aug 04 18:00:31.190336 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:31.190303    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:31.191067 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:31.190436    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:31.191067 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:31.190488    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:31.191067 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:31.190604    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:31.299141 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:31.299110    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:31.314678 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:31.314650    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:00:32.188638 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:32.188601    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:32.188887 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:32.188709    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:32.248919 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:32.248884    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t46pw"]
Aug 04 18:00:32.252096 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:32.252073    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bd4m4"]
Aug 04 18:00:32.252249 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:32.252175    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:32.252325 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:32.252288    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:32.252903 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:32.252880    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-647mg"]
Aug 04 18:00:32.253000 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:32.252979    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:32.253093 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:32.253077    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:32.302992 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:32.302964    2566 generic.go:356] "Generic (PLEG): container finished" podID="2959820d-b173-49ec-875d-9dfa00f47f50" containerID="eeddc726c7240e35381d735e0f837dfc56c5e4bd616a10726d7eb17a02f99c7b" exitCode=0
Aug 04 18:00:32.303148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:32.303053    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hmzpt" event={"ID":"2959820d-b173-49ec-875d-9dfa00f47f50","Type":"ContainerDied","Data":"eeddc726c7240e35381d735e0f837dfc56c5e4bd616a10726d7eb17a02f99c7b"}
Aug 04 18:00:32.303148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:32.303092    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:32.303229 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:32.303193    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:32.332104 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:32.332079    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v69qf_703eadd7-667a-4879-aa4e-2ccedb5b7c8a/dns-node-resolver/0.log"
Aug 04 18:00:32.909856 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:32.909831    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7f2bk_7dce6b12-1695-48e9-b13a-0bf64d3107ff/node-ca/0.log"
Aug 04 18:00:34.189592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:34.189476    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:34.189592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:34.189496    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:34.189592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:34.189587    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:34.190191 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:34.189604    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:34.190191 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:34.189683    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:34.190191 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:34.189754    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:34.310076 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:34.310042    2566 generic.go:356] "Generic (PLEG): container finished" podID="2959820d-b173-49ec-875d-9dfa00f47f50" containerID="56de568be1dfd3c31d462136eb1f2cec936cec2cec341f54f952438290319fe6" exitCode=0
Aug 04 18:00:34.310217 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:34.310104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hmzpt" event={"ID":"2959820d-b173-49ec-875d-9dfa00f47f50","Type":"ContainerDied","Data":"56de568be1dfd3c31d462136eb1f2cec936cec2cec341f54f952438290319fe6"}
Aug 04 18:00:34.827281 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:34.827242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret\") pod \"global-pull-secret-syncer-t46pw\" (UID: \"6cbc3f49-e03c-425d-9da2-71eb15915d42\") " pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:34.827498 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:34.827409    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:34.827498 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:34.827486    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret podName:6cbc3f49-e03c-425d-9da2-71eb15915d42 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:06.827469327 +0000 UTC m=+66.118083479 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret") pod "global-pull-secret-syncer-t46pw" (UID: "6cbc3f49-e03c-425d-9da2-71eb15915d42") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:34.928309 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:34.928266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs\") pod \"network-metrics-daemon-647mg\" (UID: \"47e059c0-7685-4231-86b6-5dbea4f4d5d5\") " pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:34.928485 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:34.928464    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:34.928560 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:34.928547    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs podName:47e059c0-7685-4231-86b6-5dbea4f4d5d5 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:06.928524675 +0000 UTC m=+66.219138831 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs") pod "network-metrics-daemon-647mg" (UID: "47e059c0-7685-4231-86b6-5dbea4f4d5d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:35.029198 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:35.029167    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27k4v\" (UniqueName: \"kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v\") pod \"network-check-target-bd4m4\" (UID: \"dbbadf1e-6e33-48b0-8d9c-437c756cb1eb\") " pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:35.029382 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:35.029344    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:35.029437 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:35.029388    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:35.029437 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:35.029400    2566 projected.go:196] Error preparing data for projected volume kube-api-access-27k4v for pod openshift-network-diagnostics/network-check-target-bd4m4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:35.029536 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:35.029459    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v podName:dbbadf1e-6e33-48b0-8d9c-437c756cb1eb nodeName:}" failed. No retries permitted until 2026-08-04 18:01:07.029441503 +0000 UTC m=+66.320055657 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-27k4v" (UniqueName: "kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v") pod "network-check-target-bd4m4" (UID: "dbbadf1e-6e33-48b0-8d9c-437c756cb1eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:36.189052 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:36.188970    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:36.189535 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:36.189092    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:36.189535 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:36.189100    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:36.189535 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:36.189120    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:36.189535 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:36.189210    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:36.189535 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:36.189313    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:38.188965 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:38.188928    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:38.188965 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:38.188959    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:38.189513 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:38.188961    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:38.189513 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:38.189042    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:38.189513 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:38.189109    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:38.189513 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:38.189196    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:39.324407 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:39.324374    2566 generic.go:356] "Generic (PLEG): container finished" podID="2959820d-b173-49ec-875d-9dfa00f47f50" containerID="02b1cc9eef56a5859e0b90618f9cc36526b656d89ebec258beca4dcdedf65692" exitCode=0
Aug 04 18:00:39.324850 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:39.324423    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hmzpt" event={"ID":"2959820d-b173-49ec-875d-9dfa00f47f50","Type":"ContainerDied","Data":"02b1cc9eef56a5859e0b90618f9cc36526b656d89ebec258beca4dcdedf65692"}
Aug 04 18:00:40.189523 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:40.189492    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:40.189523 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:40.189509    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:40.189723 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:40.189594    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:40.189723 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:40.189595    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:40.189723 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:40.189666    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:40.189816 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:40.189737    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:40.329655 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:40.329613    2566 generic.go:356] "Generic (PLEG): container finished" podID="2959820d-b173-49ec-875d-9dfa00f47f50" containerID="937e90df2d8f8f275a1758c5bde282f84e34f8d952c14162dde7f710c97e8a8b" exitCode=0
Aug 04 18:00:40.330026 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:40.329659    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hmzpt" event={"ID":"2959820d-b173-49ec-875d-9dfa00f47f50","Type":"ContainerDied","Data":"937e90df2d8f8f275a1758c5bde282f84e34f8d952c14162dde7f710c97e8a8b"}
Aug 04 18:00:41.335215 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:41.335024    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hmzpt" event={"ID":"2959820d-b173-49ec-875d-9dfa00f47f50","Type":"ContainerStarted","Data":"947037ec923f406c36bef181b3e563aa2c34668a89cc70846b0c8faf6c1a9b55"}
Aug 04 18:00:41.362928 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:41.362874    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hmzpt" podStartSLOduration=8.372738099 podStartE2EDuration="40.362859774s" totalImagesPullingTime="31.990121675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:41.361306645 +0000 UTC m=+40.651920816" watchObservedRunningTime="2026-08-04 18:00:41.362859774 +0000 UTC m=+40.653474169"
Aug 04 18:00:42.189081 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:42.189052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:42.189232 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:42.189052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:42.189232 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:42.189157    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:42.189232 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:42.189216    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:42.189232 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:42.189052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:42.189498 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:42.189331    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:44.189099 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:44.189062    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:44.189708 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:44.189169    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:44.189708 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:44.189181    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:44.189708 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:44.189199    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:44.189708 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:44.189281    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:44.189708 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:44.189365    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:46.189271 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:46.189232    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:46.189654 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:46.189236    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:46.189654 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:46.189331    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:46.189654 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:46.189373    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:46.189654 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:46.189458    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:46.189654 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:46.189531    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:48.188803 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:48.188778    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:48.189339 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:48.188875    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:48.189339 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:48.188897    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:48.189339 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:48.188889    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:48.189339 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:48.188953    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:48.189339 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:48.189015    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:50.189342 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:50.189306    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:50.189342 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:50.189342    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:50.189808 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:50.189417    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:50.189808 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:50.189432    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:50.189808 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:50.189498    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:50.189808 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:50.189590    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:52.188826 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:52.188785    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:52.188826 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:52.188830    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:52.189269 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:52.188883    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:52.189269 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:52.188897    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:52.189269 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:52.188960    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:52.189269 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:52.189022    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:54.189195 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:54.189163    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:54.189574 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:54.189163    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:54.189574 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:54.189288    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-647mg" podUID="47e059c0-7685-4231-86b6-5dbea4f4d5d5"
Aug 04 18:00:54.189574 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:54.189163    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:54.189574 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:54.189336    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4m4" podUID="dbbadf1e-6e33-48b0-8d9c-437c756cb1eb"
Aug 04 18:00:54.189574 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:00:54.189419    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t46pw" podUID="6cbc3f49-e03c-425d-9da2-71eb15915d42"
Aug 04 18:00:55.590656 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.590619    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-242.ec2.internal" event="NodeReady"
Aug 04 18:00:55.591071 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.590746    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Aug 04 18:00:55.651113 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.651077    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xk4xc"]
Aug 04 18:00:55.664475 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.664453    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dkvs8"]
Aug 04 18:00:55.664590 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.664574    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xk4xc"
Aug 04 18:00:55.667592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.667566    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Aug 04 18:00:55.671274 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.671252    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Aug 04 18:00:55.674517 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.674496    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g9rrt\""
Aug 04 18:00:55.674636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.674545    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xk4xc"]
Aug 04 18:00:55.674636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.674584    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dkvs8"]
Aug 04 18:00:55.674728 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.674687    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:55.676985 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.676964    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Aug 04 18:00:55.676985 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.676977    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Aug 04 18:00:55.677169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.677001    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Aug 04 18:00:55.677169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.677055    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Aug 04 18:00:55.677169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.677129    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8f16e23-a10c-4e3b-91e9-e14582c60d00-tmp-dir\") pod \"dns-default-xk4xc\" (UID: \"f8f16e23-a10c-4e3b-91e9-e14582c60d00\") " pod="openshift-dns/dns-default-xk4xc"
Aug 04 18:00:55.677169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.677162    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8f16e23-a10c-4e3b-91e9-e14582c60d00-metrics-tls\") pod \"dns-default-xk4xc\" (UID: \"f8f16e23-a10c-4e3b-91e9-e14582c60d00\") " pod="openshift-dns/dns-default-xk4xc"
Aug 04 18:00:55.677395 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.677230    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqbm8\" (UniqueName: \"kubernetes.io/projected/f8f16e23-a10c-4e3b-91e9-e14582c60d00-kube-api-access-gqbm8\") pod \"dns-default-xk4xc\" (UID: \"f8f16e23-a10c-4e3b-91e9-e14582c60d00\") " pod="openshift-dns/dns-default-xk4xc"
Aug 04 18:00:55.677395 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.677286    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f16e23-a10c-4e3b-91e9-e14582c60d00-config-volume\") pod \"dns-default-xk4xc\" (UID: \"f8f16e23-a10c-4e3b-91e9-e14582c60d00\") " pod="openshift-dns/dns-default-xk4xc"
Aug 04 18:00:55.677395 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.677297    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n9fk8\""
Aug 04 18:00:55.742684 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.742647    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fpdw5"]
Aug 04 18:00:55.748751 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.748732    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fpdw5"
Aug 04 18:00:55.751213 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.751175    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Aug 04 18:00:55.751332 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.751262    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fmq84\""
Aug 04 18:00:55.751332 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.751285    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Aug 04 18:00:55.751481 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.751392    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Aug 04 18:00:55.757082 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.757058    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fpdw5"]
Aug 04 18:00:55.778096 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.778070    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d9301b5-6d5a-4e67-9bfd-cfc549bedf26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkvs8\" (UID: \"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26\") " pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:55.778245 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.778116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d9301b5-6d5a-4e67-9bfd-cfc549bedf26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkvs8\" (UID: \"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26\") " pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:55.778245 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.778138    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f16e23-a10c-4e3b-91e9-e14582c60d00-config-volume\") pod \"dns-default-xk4xc\" (UID: \"f8f16e23-a10c-4e3b-91e9-e14582c60d00\") " pod="openshift-dns/dns-default-xk4xc"
Aug 04 18:00:55.778245 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.778203    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8mgf\" (UniqueName: \"kubernetes.io/projected/ec4a4cad-5eab-4f95-99ba-a918bfed16c9-kube-api-access-j8mgf\") pod \"ingress-canary-fpdw5\" (UID: \"ec4a4cad-5eab-4f95-99ba-a918bfed16c9\") " pod="openshift-ingress-canary/ingress-canary-fpdw5"
Aug 04 18:00:55.778245 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.778233    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d9301b5-6d5a-4e67-9bfd-cfc549bedf26-crio-socket\") pod \"insights-runtime-extractor-dkvs8\" (UID: \"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26\") " pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:55.778481 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.778266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8f16e23-a10c-4e3b-91e9-e14582c60d00-tmp-dir\") pod \"dns-default-xk4xc\" (UID: \"f8f16e23-a10c-4e3b-91e9-e14582c60d00\") " pod="openshift-dns/dns-default-xk4xc"
Aug 04 18:00:55.778481 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.778308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8f16e23-a10c-4e3b-91e9-e14582c60d00-metrics-tls\") pod \"dns-default-xk4xc\" (UID: \"f8f16e23-a10c-4e3b-91e9-e14582c60d00\") " pod="openshift-dns/dns-default-xk4xc"
Aug 04 18:00:55.778481 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.778343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d9301b5-6d5a-4e67-9bfd-cfc549bedf26-data-volume\") pod \"insights-runtime-extractor-dkvs8\" (UID: \"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26\") " pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:55.778481 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.778376    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffswq\" (UniqueName: \"kubernetes.io/projected/8d9301b5-6d5a-4e67-9bfd-cfc549bedf26-kube-api-access-ffswq\") pod \"insights-runtime-extractor-dkvs8\" (UID: \"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26\") " pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:55.778481 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.778446    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqbm8\" (UniqueName: \"kubernetes.io/projected/f8f16e23-a10c-4e3b-91e9-e14582c60d00-kube-api-access-gqbm8\") pod \"dns-default-xk4xc\" (UID: \"f8f16e23-a10c-4e3b-91e9-e14582c60d00\") " pod="openshift-dns/dns-default-xk4xc"
Aug 04 18:00:55.778715 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.778577    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8f16e23-a10c-4e3b-91e9-e14582c60d00-tmp-dir\") pod \"dns-default-xk4xc\" (UID: \"f8f16e23-a10c-4e3b-91e9-e14582c60d00\") " pod="openshift-dns/dns-default-xk4xc"
Aug 04 18:00:55.778715 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.778588    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f16e23-a10c-4e3b-91e9-e14582c60d00-config-volume\") pod \"dns-default-xk4xc\" (UID: \"f8f16e23-a10c-4e3b-91e9-e14582c60d00\") " pod="openshift-dns/dns-default-xk4xc"
Aug 04 18:00:55.778715 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.778621    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec4a4cad-5eab-4f95-99ba-a918bfed16c9-cert\") pod \"ingress-canary-fpdw5\" (UID: \"ec4a4cad-5eab-4f95-99ba-a918bfed16c9\") " pod="openshift-ingress-canary/ingress-canary-fpdw5"
Aug 04 18:00:55.782416 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.782397    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8f16e23-a10c-4e3b-91e9-e14582c60d00-metrics-tls\") pod \"dns-default-xk4xc\" (UID: \"f8f16e23-a10c-4e3b-91e9-e14582c60d00\") " pod="openshift-dns/dns-default-xk4xc"
Aug 04 18:00:55.786710 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.786691    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqbm8\" (UniqueName: \"kubernetes.io/projected/f8f16e23-a10c-4e3b-91e9-e14582c60d00-kube-api-access-gqbm8\") pod \"dns-default-xk4xc\" (UID: \"f8f16e23-a10c-4e3b-91e9-e14582c60d00\") " pod="openshift-dns/dns-default-xk4xc"
Aug 04 18:00:55.879991 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.879911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec4a4cad-5eab-4f95-99ba-a918bfed16c9-cert\") pod \"ingress-canary-fpdw5\" (UID: \"ec4a4cad-5eab-4f95-99ba-a918bfed16c9\") " pod="openshift-ingress-canary/ingress-canary-fpdw5"
Aug 04 18:00:55.879991 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.879953    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d9301b5-6d5a-4e67-9bfd-cfc549bedf26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkvs8\" (UID: \"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26\") " pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:55.880196 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.879999    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d9301b5-6d5a-4e67-9bfd-cfc549bedf26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkvs8\" (UID: \"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26\") " pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:55.880196 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.880034    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8mgf\" (UniqueName: \"kubernetes.io/projected/ec4a4cad-5eab-4f95-99ba-a918bfed16c9-kube-api-access-j8mgf\") pod \"ingress-canary-fpdw5\" (UID: \"ec4a4cad-5eab-4f95-99ba-a918bfed16c9\") " pod="openshift-ingress-canary/ingress-canary-fpdw5"
Aug 04 18:00:55.880196 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.880184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d9301b5-6d5a-4e67-9bfd-cfc549bedf26-crio-socket\") pod \"insights-runtime-extractor-dkvs8\" (UID: \"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26\") " pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:55.880409 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.880223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d9301b5-6d5a-4e67-9bfd-cfc549bedf26-data-volume\") pod \"insights-runtime-extractor-dkvs8\" (UID: \"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26\") " pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:55.880409 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.880248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffswq\" (UniqueName: \"kubernetes.io/projected/8d9301b5-6d5a-4e67-9bfd-cfc549bedf26-kube-api-access-ffswq\") pod \"insights-runtime-extractor-dkvs8\" (UID: \"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26\") " pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:55.880533 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.880460    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d9301b5-6d5a-4e67-9bfd-cfc549bedf26-crio-socket\") pod \"insights-runtime-extractor-dkvs8\" (UID: \"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26\") " pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:55.880600 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.880550    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d9301b5-6d5a-4e67-9bfd-cfc549bedf26-data-volume\") pod \"insights-runtime-extractor-dkvs8\" (UID: \"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26\") " pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:55.880736 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.880718    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d9301b5-6d5a-4e67-9bfd-cfc549bedf26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkvs8\" (UID: \"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26\") " pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:55.882418 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.882392    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec4a4cad-5eab-4f95-99ba-a918bfed16c9-cert\") pod \"ingress-canary-fpdw5\" (UID: \"ec4a4cad-5eab-4f95-99ba-a918bfed16c9\") " pod="openshift-ingress-canary/ingress-canary-fpdw5"
Aug 04 18:00:55.882418 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.882395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d9301b5-6d5a-4e67-9bfd-cfc549bedf26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkvs8\" (UID: \"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26\") " pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:55.888416 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.888395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8mgf\" (UniqueName: \"kubernetes.io/projected/ec4a4cad-5eab-4f95-99ba-a918bfed16c9-kube-api-access-j8mgf\") pod \"ingress-canary-fpdw5\" (UID: \"ec4a4cad-5eab-4f95-99ba-a918bfed16c9\") " pod="openshift-ingress-canary/ingress-canary-fpdw5"
Aug 04 18:00:55.888491 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.888440    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffswq\" (UniqueName: \"kubernetes.io/projected/8d9301b5-6d5a-4e67-9bfd-cfc549bedf26-kube-api-access-ffswq\") pod \"insights-runtime-extractor-dkvs8\" (UID: \"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26\") " pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:55.974496 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.974457    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xk4xc"
Aug 04 18:00:55.982719 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:55.982698    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dkvs8"
Aug 04 18:00:56.057481 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.057454    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fpdw5"
Aug 04 18:00:56.184373 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.182402    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xk4xc"]
Aug 04 18:00:56.191393 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.190766    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:00:56.191393 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.191300    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:00:56.191623 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.191477    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:00:56.196559 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.195411    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 18:00:56.196559 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.195708    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 18:00:56.196559 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.195894    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h457s\""
Aug 04 18:00:56.196559 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.195974    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 18:00:56.196559 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.196138    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 18:00:56.196559 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.196309    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5s4h\""
Aug 04 18:00:56.200836 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:56.200513    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f16e23_a10c_4e3b_91e9_e14582c60d00.slice/crio-1cea309fab1592d801e5b3f7eba6e9725e8094e9e7f84ee899098facf325ebd0 WatchSource:0}: Error finding container 1cea309fab1592d801e5b3f7eba6e9725e8094e9e7f84ee899098facf325ebd0: Status 404 returned error can't find the container with id 1cea309fab1592d801e5b3f7eba6e9725e8094e9e7f84ee899098facf325ebd0
Aug 04 18:00:56.224292 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.224260    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dkvs8"]
Aug 04 18:00:56.229956 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:56.229926    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9301b5_6d5a_4e67_9bfd_cfc549bedf26.slice/crio-e645ac60074edf67f9e3910c7290955b9df60b8ad6d0511406732fdf5c7dd587 WatchSource:0}: Error finding container e645ac60074edf67f9e3910c7290955b9df60b8ad6d0511406732fdf5c7dd587: Status 404 returned error can't find the container with id e645ac60074edf67f9e3910c7290955b9df60b8ad6d0511406732fdf5c7dd587
Aug 04 18:00:56.246848 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.246819    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fpdw5"]
Aug 04 18:00:56.250109 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:00:56.250084    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4a4cad_5eab_4f95_99ba_a918bfed16c9.slice/crio-22c4b73edb8436f07f32dace77e6f2cfdfc082abebda87f256997e170286df5a WatchSource:0}: Error finding container 22c4b73edb8436f07f32dace77e6f2cfdfc082abebda87f256997e170286df5a: Status 404 returned error can't find the container with id 22c4b73edb8436f07f32dace77e6f2cfdfc082abebda87f256997e170286df5a
Aug 04 18:00:56.368619 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.368585    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dkvs8" event={"ID":"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26","Type":"ContainerStarted","Data":"27a6db4340eddcfe35f631f771483192df1a90ce97460353ad1f772435ea4f99"}
Aug 04 18:00:56.368796 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.368644    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dkvs8" event={"ID":"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26","Type":"ContainerStarted","Data":"e645ac60074edf67f9e3910c7290955b9df60b8ad6d0511406732fdf5c7dd587"}
Aug 04 18:00:56.369642 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.369620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fpdw5" event={"ID":"ec4a4cad-5eab-4f95-99ba-a918bfed16c9","Type":"ContainerStarted","Data":"22c4b73edb8436f07f32dace77e6f2cfdfc082abebda87f256997e170286df5a"}
Aug 04 18:00:56.370512 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:56.370496    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xk4xc" event={"ID":"f8f16e23-a10c-4e3b-91e9-e14582c60d00","Type":"ContainerStarted","Data":"1cea309fab1592d801e5b3f7eba6e9725e8094e9e7f84ee899098facf325ebd0"}
Aug 04 18:00:58.380381 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:58.380326    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dkvs8" event={"ID":"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26","Type":"ContainerStarted","Data":"3b515a965ae34b041a05f8ae693dc00f3e163ca60d77f81d0424e09833c90990"}
Aug 04 18:00:59.384671 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:59.384610    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fpdw5" event={"ID":"ec4a4cad-5eab-4f95-99ba-a918bfed16c9","Type":"ContainerStarted","Data":"a5f5ab8431a53d939338c3b6971b69327ceb1d1927b3e23d0c869545f6dfca46"}
Aug 04 18:00:59.407766 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:00:59.407710    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fpdw5" podStartSLOduration=1.6796248550000001 podStartE2EDuration="4.407692947s" totalImagesPullingTime="2.728068092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:59.407545972 +0000 UTC m=+58.698160154" watchObservedRunningTime="2026-08-04 18:00:59.407692947 +0000 UTC m=+58.698307120"
Aug 04 18:01:00.673911 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:00.673887    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vrdvt"]
Aug 04 18:01:00.676376 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:00.676330    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vrdvt"
Aug 04 18:01:00.679055 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:00.678840    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Aug 04 18:01:00.679055 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:00.678973    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-69spp\""
Aug 04 18:01:00.685545 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:00.685419    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vrdvt"]
Aug 04 18:01:00.718267 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:00.718244    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/49a382ef-5436-4392-a0ac-327454be1c2d-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-vrdvt\" (UID: \"49a382ef-5436-4392-a0ac-327454be1c2d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vrdvt"
Aug 04 18:01:00.819407 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:00.819384    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/49a382ef-5436-4392-a0ac-327454be1c2d-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-vrdvt\" (UID: \"49a382ef-5436-4392-a0ac-327454be1c2d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vrdvt"
Aug 04 18:01:00.821553 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:00.821529    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/49a382ef-5436-4392-a0ac-327454be1c2d-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-vrdvt\" (UID: \"49a382ef-5436-4392-a0ac-327454be1c2d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vrdvt"
Aug 04 18:01:01.005809 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:01.005716    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vrdvt"
Aug 04 18:01:01.131515 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:01.131475    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vrdvt"]
Aug 04 18:01:01.135835 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:01:01.135807    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a382ef_5436_4392_a0ac_327454be1c2d.slice/crio-ca3e511e6a5b02fc93efdd094017f1eb1c9a34decce5a85d26a5088a19dc330f WatchSource:0}: Error finding container ca3e511e6a5b02fc93efdd094017f1eb1c9a34decce5a85d26a5088a19dc330f: Status 404 returned error can't find the container with id ca3e511e6a5b02fc93efdd094017f1eb1c9a34decce5a85d26a5088a19dc330f
Aug 04 18:01:01.391522 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:01.391482    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dkvs8" event={"ID":"8d9301b5-6d5a-4e67-9bfd-cfc549bedf26","Type":"ContainerStarted","Data":"55e27457cdbd198034706c7abc9113d84ffa0f4512cc60ce52754e91a9449d1b"}
Aug 04 18:01:01.393193 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:01.393167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xk4xc" event={"ID":"f8f16e23-a10c-4e3b-91e9-e14582c60d00","Type":"ContainerStarted","Data":"c4634485b4a1200dcd2e1b934cd7d75b4b98c405c8c9d09e3aa3a6d83908f57b"}
Aug 04 18:01:01.393341 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:01.393220    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xk4xc"
Aug 04 18:01:01.393341 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:01.393247    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xk4xc" event={"ID":"f8f16e23-a10c-4e3b-91e9-e14582c60d00","Type":"ContainerStarted","Data":"ebdd132ade511e81aae4d392f809046f8e6a57739d45d91121b05573237d2e3b"}
Aug 04 18:01:01.394270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:01.394246    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vrdvt" event={"ID":"49a382ef-5436-4392-a0ac-327454be1c2d","Type":"ContainerStarted","Data":"ca3e511e6a5b02fc93efdd094017f1eb1c9a34decce5a85d26a5088a19dc330f"}
Aug 04 18:01:01.411242 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:01.411190    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dkvs8" podStartSLOduration=2.157939294 podStartE2EDuration="6.411176403s" totalImagesPullingTime="4.253237109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:01.409859373 +0000 UTC m=+60.700473567" watchObservedRunningTime="2026-08-04 18:01:01.411176403 +0000 UTC m=+60.701790552"
Aug 04 18:01:01.428146 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:01.428099    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xk4xc" podStartSLOduration=1.969124798 podStartE2EDuration="6.428086018s" totalImagesPullingTime="4.45896122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:01.427929305 +0000 UTC m=+60.718543500" watchObservedRunningTime="2026-08-04 18:01:01.428086018 +0000 UTC m=+60.718700166"
Aug 04 18:01:02.399048 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:02.399008    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vrdvt" event={"ID":"49a382ef-5436-4392-a0ac-327454be1c2d","Type":"ContainerStarted","Data":"57ee99d6166dd30a87cf9ef78ee269495cfb0ac9e532ea3e9bf0c91b22bf00e0"}
Aug 04 18:01:02.418534 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:02.418484    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vrdvt" podStartSLOduration=1.215007156 podStartE2EDuration="2.418468107s" totalImagesPullingTime="1.203460951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:02.417644411 +0000 UTC m=+61.708258598" watchObservedRunningTime="2026-08-04 18:01:02.418468107 +0000 UTC m=+61.709082278"
Aug 04 18:01:03.319054 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.319023    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfzm2"
Aug 04 18:01:03.402513 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.402483    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vrdvt"
Aug 04 18:01:03.407326 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.407303    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vrdvt"
Aug 04 18:01:03.749124 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.749041    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-xb49f"]
Aug 04 18:01:03.760886 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.760854    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68bfcb7466-h97jl"]
Aug 04 18:01:03.761047 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.760971    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f"
Aug 04 18:01:03.763908 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.763640    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Aug 04 18:01:03.763908 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.763744    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Aug 04 18:01:03.764141 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.764120    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Aug 04 18:01:03.764243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.764230    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Aug 04 18:01:03.764367 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.764337    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ql7dn\""
Aug 04 18:01:03.764494 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.764474    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Aug 04 18:01:03.787043 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.787017    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-xb49f"]
Aug 04 18:01:03.787156 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.787059    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68bfcb7466-h97jl"]
Aug 04 18:01:03.787156 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.787130    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.789881 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.789864    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Aug 04 18:01:03.790010 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.789905    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hkgfc\""
Aug 04 18:01:03.790010 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.789967    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Aug 04 18:01:03.790125 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.790110    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Aug 04 18:01:03.790293 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.790276    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Aug 04 18:01:03.790414 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.790373    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Aug 04 18:01:03.790484 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.790411    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Aug 04 18:01:03.791269 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.791247    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Aug 04 18:01:03.843045 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.843015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45ea4544-fff5-40fd-9fa3-60fc77126170-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-xb49f\" (UID: \"45ea4544-fff5-40fd-9fa3-60fc77126170\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f"
Aug 04 18:01:03.843045 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.843049    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-service-ca\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.843231 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.843068    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-console-config\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.843231 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.843089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdzrw\" (UniqueName: \"kubernetes.io/projected/db17423e-c693-4630-a50d-d8b5567f2051-kube-api-access-zdzrw\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.843231 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.843115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45ea4544-fff5-40fd-9fa3-60fc77126170-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-xb49f\" (UID: \"45ea4544-fff5-40fd-9fa3-60fc77126170\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f"
Aug 04 18:01:03.843231 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.843167    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db17423e-c693-4630-a50d-d8b5567f2051-console-serving-cert\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.843231 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.843202    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db17423e-c693-4630-a50d-d8b5567f2051-console-oauth-config\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.843231 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.843223    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5pq7\" (UniqueName: \"kubernetes.io/projected/45ea4544-fff5-40fd-9fa3-60fc77126170-kube-api-access-r5pq7\") pod \"prometheus-operator-758d7cb874-xb49f\" (UID: \"45ea4544-fff5-40fd-9fa3-60fc77126170\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f"
Aug 04 18:01:03.843594 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.843246    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-oauth-serving-cert\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.843594 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.843268    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/45ea4544-fff5-40fd-9fa3-60fc77126170-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-xb49f\" (UID: \"45ea4544-fff5-40fd-9fa3-60fc77126170\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f"
Aug 04 18:01:03.943990 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.943961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-oauth-serving-cert\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.943990 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.943993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/45ea4544-fff5-40fd-9fa3-60fc77126170-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-xb49f\" (UID: \"45ea4544-fff5-40fd-9fa3-60fc77126170\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f"
Aug 04 18:01:03.944204 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.944034    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45ea4544-fff5-40fd-9fa3-60fc77126170-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-xb49f\" (UID: \"45ea4544-fff5-40fd-9fa3-60fc77126170\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f"
Aug 04 18:01:03.944204 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.944065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-service-ca\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.944204 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.944094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-console-config\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.944204 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.944126    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzrw\" (UniqueName: \"kubernetes.io/projected/db17423e-c693-4630-a50d-d8b5567f2051-kube-api-access-zdzrw\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.944204 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.944147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45ea4544-fff5-40fd-9fa3-60fc77126170-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-xb49f\" (UID: \"45ea4544-fff5-40fd-9fa3-60fc77126170\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f"
Aug 04 18:01:03.944204 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.944200    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db17423e-c693-4630-a50d-d8b5567f2051-console-serving-cert\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.944534 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.944245    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db17423e-c693-4630-a50d-d8b5567f2051-console-oauth-config\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.944534 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.944272    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5pq7\" (UniqueName: \"kubernetes.io/projected/45ea4544-fff5-40fd-9fa3-60fc77126170-kube-api-access-r5pq7\") pod \"prometheus-operator-758d7cb874-xb49f\" (UID: \"45ea4544-fff5-40fd-9fa3-60fc77126170\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f"
Aug 04 18:01:03.944950 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.944922    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-console-config\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.945058 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.944927    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-oauth-serving-cert\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.945058 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.944938    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45ea4544-fff5-40fd-9fa3-60fc77126170-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-xb49f\" (UID: \"45ea4544-fff5-40fd-9fa3-60fc77126170\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f"
Aug 04 18:01:03.945396 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.945301    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-service-ca\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.946810 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.946785    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/45ea4544-fff5-40fd-9fa3-60fc77126170-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-xb49f\" (UID: \"45ea4544-fff5-40fd-9fa3-60fc77126170\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f"
Aug 04 18:01:03.946885 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.946816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45ea4544-fff5-40fd-9fa3-60fc77126170-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-xb49f\" (UID: \"45ea4544-fff5-40fd-9fa3-60fc77126170\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f"
Aug 04 18:01:03.947065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.947045    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db17423e-c693-4630-a50d-d8b5567f2051-console-serving-cert\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.947116 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.947102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db17423e-c693-4630-a50d-d8b5567f2051-console-oauth-config\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:03.954788 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.954762    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5pq7\" (UniqueName: \"kubernetes.io/projected/45ea4544-fff5-40fd-9fa3-60fc77126170-kube-api-access-r5pq7\") pod \"prometheus-operator-758d7cb874-xb49f\" (UID: \"45ea4544-fff5-40fd-9fa3-60fc77126170\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f"
Aug 04 18:01:03.955913 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:03.955895    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdzrw\" (UniqueName: \"kubernetes.io/projected/db17423e-c693-4630-a50d-d8b5567f2051-kube-api-access-zdzrw\") pod \"console-68bfcb7466-h97jl\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") " pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:04.071676 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:04.071638    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f"
Aug 04 18:01:04.095886 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:04.095855    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:04.231993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:04.231938    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-xb49f"]
Aug 04 18:01:04.234270 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:01:04.234246    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ea4544_fff5_40fd_9fa3_60fc77126170.slice/crio-5da2454ab3c7a173143176bb30593a304f95f31aa8f02278c8213edffffadeb2 WatchSource:0}: Error finding container 5da2454ab3c7a173143176bb30593a304f95f31aa8f02278c8213edffffadeb2: Status 404 returned error can't find the container with id 5da2454ab3c7a173143176bb30593a304f95f31aa8f02278c8213edffffadeb2
Aug 04 18:01:04.249099 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:04.249075    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68bfcb7466-h97jl"]
Aug 04 18:01:04.252436 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:01:04.252410    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb17423e_c693_4630_a50d_d8b5567f2051.slice/crio-c2bbce2562801143fcf4f79365a888199ea07f546f2316110a39d8940989d94e WatchSource:0}: Error finding container c2bbce2562801143fcf4f79365a888199ea07f546f2316110a39d8940989d94e: Status 404 returned error can't find the container with id c2bbce2562801143fcf4f79365a888199ea07f546f2316110a39d8940989d94e
Aug 04 18:01:04.407406 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:04.407302    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f" event={"ID":"45ea4544-fff5-40fd-9fa3-60fc77126170","Type":"ContainerStarted","Data":"5da2454ab3c7a173143176bb30593a304f95f31aa8f02278c8213edffffadeb2"}
Aug 04 18:01:04.408499 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:04.408474    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68bfcb7466-h97jl" event={"ID":"db17423e-c693-4630-a50d-d8b5567f2051","Type":"ContainerStarted","Data":"c2bbce2562801143fcf4f79365a888199ea07f546f2316110a39d8940989d94e"}
Aug 04 18:01:06.872423 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:06.872383    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret\") pod \"global-pull-secret-syncer-t46pw\" (UID: \"6cbc3f49-e03c-425d-9da2-71eb15915d42\") " pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:01:06.875147 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:06.875115    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 18:01:06.885572 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:06.885543    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6cbc3f49-e03c-425d-9da2-71eb15915d42-original-pull-secret\") pod \"global-pull-secret-syncer-t46pw\" (UID: \"6cbc3f49-e03c-425d-9da2-71eb15915d42\") " pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:01:06.973141 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:06.973107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs\") pod \"network-metrics-daemon-647mg\" (UID: \"47e059c0-7685-4231-86b6-5dbea4f4d5d5\") " pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:01:06.976138 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:06.976113    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 18:01:06.986159 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:06.986126    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47e059c0-7685-4231-86b6-5dbea4f4d5d5-metrics-certs\") pod \"network-metrics-daemon-647mg\" (UID: \"47e059c0-7685-4231-86b6-5dbea4f4d5d5\") " pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:01:07.016058 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:07.016022    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t46pw"
Aug 04 18:01:07.023587 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:07.023561    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5s4h\""
Aug 04 18:01:07.031636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:07.031614    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-647mg"
Aug 04 18:01:07.074083 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:07.074047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27k4v\" (UniqueName: \"kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v\") pod \"network-check-target-bd4m4\" (UID: \"dbbadf1e-6e33-48b0-8d9c-437c756cb1eb\") " pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:01:07.076636 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:07.076606    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 18:01:07.086744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:07.086720    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 18:01:07.098270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:07.098242    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27k4v\" (UniqueName: \"kubernetes.io/projected/dbbadf1e-6e33-48b0-8d9c-437c756cb1eb-kube-api-access-27k4v\") pod \"network-check-target-bd4m4\" (UID: \"dbbadf1e-6e33-48b0-8d9c-437c756cb1eb\") " pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:01:07.311113 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:07.311089    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h457s\""
Aug 04 18:01:07.318588 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:07.318509    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:01:07.373014 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:07.372936    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t46pw"]
Aug 04 18:01:07.377899 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:01:07.377827    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbc3f49_e03c_425d_9da2_71eb15915d42.slice/crio-90e369435c7324a837ea8995c757adcd51b9084f2f50e04b9f3750d6bd3026a2 WatchSource:0}: Error finding container 90e369435c7324a837ea8995c757adcd51b9084f2f50e04b9f3750d6bd3026a2: Status 404 returned error can't find the container with id 90e369435c7324a837ea8995c757adcd51b9084f2f50e04b9f3750d6bd3026a2
Aug 04 18:01:07.393767 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:07.393691    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-647mg"]
Aug 04 18:01:07.398369 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:01:07.398325    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e059c0_7685_4231_86b6_5dbea4f4d5d5.slice/crio-5c80d785d8a43e13658f3dfe6bb6b552bdb19fcb944b4c4f6e3a502d43bf22bb WatchSource:0}: Error finding container 5c80d785d8a43e13658f3dfe6bb6b552bdb19fcb944b4c4f6e3a502d43bf22bb: Status 404 returned error can't find the container with id 5c80d785d8a43e13658f3dfe6bb6b552bdb19fcb944b4c4f6e3a502d43bf22bb
Aug 04 18:01:07.425986 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:07.425914    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68bfcb7466-h97jl" event={"ID":"db17423e-c693-4630-a50d-d8b5567f2051","Type":"ContainerStarted","Data":"1acf7ecfc830abb31db72a3a389c49c33d92b99e4592fbb9bfafe764c196deff"}
Aug 04 18:01:07.428951 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:07.427752    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-647mg" event={"ID":"47e059c0-7685-4231-86b6-5dbea4f4d5d5","Type":"ContainerStarted","Data":"5c80d785d8a43e13658f3dfe6bb6b552bdb19fcb944b4c4f6e3a502d43bf22bb"}
Aug 04 18:01:07.429119 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:07.429093    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t46pw" event={"ID":"6cbc3f49-e03c-425d-9da2-71eb15915d42","Type":"ContainerStarted","Data":"90e369435c7324a837ea8995c757adcd51b9084f2f50e04b9f3750d6bd3026a2"}
Aug 04 18:01:07.446112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:07.446054    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68bfcb7466-h97jl" podStartSLOduration=1.441973781 podStartE2EDuration="4.446034312s" totalImagesPullingTime="3.004060531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:07.445751655 +0000 UTC m=+66.736365849" watchObservedRunningTime="2026-08-04 18:01:07.446034312 +0000 UTC m=+66.736648484"
Aug 04 18:01:07.472674 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:07.472606    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bd4m4"]
Aug 04 18:01:07.476546 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:01:07.476515    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbbadf1e_6e33_48b0_8d9c_437c756cb1eb.slice/crio-5768402e300403e1941b207ca1fa96b264831a7039e959796133d69d3844ce37 WatchSource:0}: Error finding container 5768402e300403e1941b207ca1fa96b264831a7039e959796133d69d3844ce37: Status 404 returned error can't find the container with id 5768402e300403e1941b207ca1fa96b264831a7039e959796133d69d3844ce37
Aug 04 18:01:08.434429 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:08.434393    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bd4m4" event={"ID":"dbbadf1e-6e33-48b0-8d9c-437c756cb1eb","Type":"ContainerStarted","Data":"5768402e300403e1941b207ca1fa96b264831a7039e959796133d69d3844ce37"}
Aug 04 18:01:08.436651 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:08.436620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f" event={"ID":"45ea4544-fff5-40fd-9fa3-60fc77126170","Type":"ContainerStarted","Data":"436bb1e755898759ae6fc83d6b906f575885634313d29e0c8297e9eb5377aa8a"}
Aug 04 18:01:08.436780 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:08.436678    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f" event={"ID":"45ea4544-fff5-40fd-9fa3-60fc77126170","Type":"ContainerStarted","Data":"7a5e36bc6dd9e9a02f9c70027031a7bd2d5868d30046031ccf2fb9aad651eb60"}
Aug 04 18:01:08.455622 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:08.455569    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-xb49f" podStartSLOduration=1.661850681 podStartE2EDuration="5.455552043s" totalImagesPullingTime="3.793701362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:08.454583662 +0000 UTC m=+67.745197864" watchObservedRunningTime="2026-08-04 18:01:08.455552043 +0000 UTC m=+67.746166213"
Aug 04 18:01:09.442611 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:09.442564    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-647mg" event={"ID":"47e059c0-7685-4231-86b6-5dbea4f4d5d5","Type":"ContainerStarted","Data":"cd15bc8edf51274dd7a44d59fe821252480b65708f17ccdc60f8e36236eab68a"}
Aug 04 18:01:09.443056 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:09.442643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-647mg" event={"ID":"47e059c0-7685-4231-86b6-5dbea4f4d5d5","Type":"ContainerStarted","Data":"4581b2798f5101cb94301987d5d576b9ac894c104ea96c4b50297f23e169f82d"}
Aug 04 18:01:09.461199 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:09.461140    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-647mg" podStartSLOduration=67.246159987 podStartE2EDuration="1m8.461121401s" totalImagesPullingTime="1.214961414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:09.460268188 +0000 UTC m=+68.750882385" watchObservedRunningTime="2026-08-04 18:01:09.461121401 +0000 UTC m=+68.751735575"
Aug 04 18:01:09.952257 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:09.952227    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57585d585c-ts8p2"]
Aug 04 18:01:09.955170 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:09.955149    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:09.963430 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:09.963403    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Aug 04 18:01:09.967215 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:09.967182    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57585d585c-ts8p2"]
Aug 04 18:01:09.995021 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:09.994986    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-trusted-ca-bundle\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:09.995167 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:09.995034    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvfkm\" (UniqueName: \"kubernetes.io/projected/416d258c-7d82-4310-84f3-dc32908b0a4e-kube-api-access-hvfkm\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:09.995167 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:09.995058    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-console-config\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:09.995167 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:09.995136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/416d258c-7d82-4310-84f3-dc32908b0a4e-console-serving-cert\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:09.995285 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:09.995173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-service-ca\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:09.995285 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:09.995201    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/416d258c-7d82-4310-84f3-dc32908b0a4e-console-oauth-config\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:09.995369 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:09.995292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-oauth-serving-cert\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:10.096690 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.096650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-trusted-ca-bundle\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:10.096877 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.096720    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvfkm\" (UniqueName: \"kubernetes.io/projected/416d258c-7d82-4310-84f3-dc32908b0a4e-kube-api-access-hvfkm\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:10.096877 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.096757    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-console-config\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:10.096877 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.096781    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/416d258c-7d82-4310-84f3-dc32908b0a4e-console-serving-cert\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:10.096877 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.096801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-service-ca\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:10.096877 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.096824    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/416d258c-7d82-4310-84f3-dc32908b0a4e-console-oauth-config\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:10.096877 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.096877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-oauth-serving-cert\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:10.097594 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.097557    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-console-config\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:10.097594 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.097588    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-service-ca\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:10.097719 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.097588    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-trusted-ca-bundle\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:10.097816 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.097792    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-oauth-serving-cert\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:10.099791 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.099765    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/416d258c-7d82-4310-84f3-dc32908b0a4e-console-oauth-config\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:10.100019 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.099995    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/416d258c-7d82-4310-84f3-dc32908b0a4e-console-serving-cert\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:10.114969 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.114948    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvfkm\" (UniqueName: \"kubernetes.io/projected/416d258c-7d82-4310-84f3-dc32908b0a4e-kube-api-access-hvfkm\") pod \"console-57585d585c-ts8p2\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") " pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:10.152780 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.152747    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"]
Aug 04 18:01:10.159954 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.159930    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"
Aug 04 18:01:10.162829 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.162802    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fdtr7\""
Aug 04 18:01:10.162829 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.162822    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Aug 04 18:01:10.163086 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.163067    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Aug 04 18:01:10.171455 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.171422    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"]
Aug 04 18:01:10.175935 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.175912    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"]
Aug 04 18:01:10.181122 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.181099    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.182206 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.182185    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k5tqn"]
Aug 04 18:01:10.184805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.184664    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Aug 04 18:01:10.184805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.184698    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Aug 04 18:01:10.185122 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.185098    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Aug 04 18:01:10.185377 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.185340    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nkzhx\""
Aug 04 18:01:10.187427 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.187387    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.190596 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.190575    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ltd8z\""
Aug 04 18:01:10.190694 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.190652    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Aug 04 18:01:10.190808 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.190789    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Aug 04 18:01:10.190891 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.190873    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Aug 04 18:01:10.191729 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.191653    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"]
Aug 04 18:01:10.198092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.197657    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0f3e4e0-1cb2-4782-a160-b2d4c530fc48-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-kh899\" (UID: \"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"
Aug 04 18:01:10.198092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.197701    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0f3e4e0-1cb2-4782-a160-b2d4c530fc48-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-kh899\" (UID: \"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"
Aug 04 18:01:10.198092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.197737    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0f3e4e0-1cb2-4782-a160-b2d4c530fc48-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-kh899\" (UID: \"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"
Aug 04 18:01:10.198092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.197790    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxxtf\" (UniqueName: \"kubernetes.io/projected/d0f3e4e0-1cb2-4782-a160-b2d4c530fc48-kube-api-access-gxxtf\") pod \"openshift-state-metrics-6bb4d7b4dd-kh899\" (UID: \"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"
Aug 04 18:01:10.265985 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.265870    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:10.298561 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.298518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de118084-296a-4dfb-b5ed-4f869a8e1b50-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.298561 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.298568    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/64a8adba-c9b7-4a79-826a-d64a1668f4cf-root\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.298824 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.298598    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.298824 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.298633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-textfile\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.298824 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.298658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/de118084-296a-4dfb-b5ed-4f869a8e1b50-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.298824 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.298680    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.298824 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.298705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0f3e4e0-1cb2-4782-a160-b2d4c530fc48-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-kh899\" (UID: \"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"
Aug 04 18:01:10.298824 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.298729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-tls\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.298824 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.298754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/de118084-296a-4dfb-b5ed-4f869a8e1b50-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.298824 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.298787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64a8adba-c9b7-4a79-826a-d64a1668f4cf-metrics-client-ca\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.299228 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.298831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98dbz\" (UniqueName: \"kubernetes.io/projected/64a8adba-c9b7-4a79-826a-d64a1668f4cf-kube-api-access-98dbz\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.299228 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.298857    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-wtmp\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.299228 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.298892    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0f3e4e0-1cb2-4782-a160-b2d4c530fc48-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-kh899\" (UID: \"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"
Aug 04 18:01:10.299228 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.298925    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de118084-296a-4dfb-b5ed-4f869a8e1b50-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.299228 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.298952    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64a8adba-c9b7-4a79-826a-d64a1668f4cf-sys\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.299228 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.298981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0f3e4e0-1cb2-4782-a160-b2d4c530fc48-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-kh899\" (UID: \"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"
Aug 04 18:01:10.299228 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.299021    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5cq8\" (UniqueName: \"kubernetes.io/projected/de118084-296a-4dfb-b5ed-4f869a8e1b50-kube-api-access-h5cq8\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.299228 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.299062    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de118084-296a-4dfb-b5ed-4f869a8e1b50-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.299228 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.299099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxxtf\" (UniqueName: \"kubernetes.io/projected/d0f3e4e0-1cb2-4782-a160-b2d4c530fc48-kube-api-access-gxxtf\") pod \"openshift-state-metrics-6bb4d7b4dd-kh899\" (UID: \"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"
Aug 04 18:01:10.299699 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:01:10.299679    2566 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Aug 04 18:01:10.299891 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:01:10.299748    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0f3e4e0-1cb2-4782-a160-b2d4c530fc48-openshift-state-metrics-tls podName:d0f3e4e0-1cb2-4782-a160-b2d4c530fc48 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:10.799727041 +0000 UTC m=+70.090341202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d0f3e4e0-1cb2-4782-a160-b2d4c530fc48-openshift-state-metrics-tls") pod "openshift-state-metrics-6bb4d7b4dd-kh899" (UID: "d0f3e4e0-1cb2-4782-a160-b2d4c530fc48") : secret "openshift-state-metrics-tls" not found
Aug 04 18:01:10.300495 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.300466    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0f3e4e0-1cb2-4782-a160-b2d4c530fc48-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-kh899\" (UID: \"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"
Aug 04 18:01:10.302844 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.302816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0f3e4e0-1cb2-4782-a160-b2d4c530fc48-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-kh899\" (UID: \"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"
Aug 04 18:01:10.315661 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.315624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxxtf\" (UniqueName: \"kubernetes.io/projected/d0f3e4e0-1cb2-4782-a160-b2d4c530fc48-kube-api-access-gxxtf\") pod \"openshift-state-metrics-6bb4d7b4dd-kh899\" (UID: \"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"
Aug 04 18:01:10.400254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.400213    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.400439 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.400287    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-tls\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.400439 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.400325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/de118084-296a-4dfb-b5ed-4f869a8e1b50-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.400439 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.400380    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64a8adba-c9b7-4a79-826a-d64a1668f4cf-metrics-client-ca\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.400439 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.400406    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98dbz\" (UniqueName: \"kubernetes.io/projected/64a8adba-c9b7-4a79-826a-d64a1668f4cf-kube-api-access-98dbz\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.400439 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.400428    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-wtmp\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.400674 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.400462    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de118084-296a-4dfb-b5ed-4f869a8e1b50-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.400674 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.400502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64a8adba-c9b7-4a79-826a-d64a1668f4cf-sys\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.400674 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.400539    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5cq8\" (UniqueName: \"kubernetes.io/projected/de118084-296a-4dfb-b5ed-4f869a8e1b50-kube-api-access-h5cq8\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.400674 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.400578    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de118084-296a-4dfb-b5ed-4f869a8e1b50-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.400674 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.400635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de118084-296a-4dfb-b5ed-4f869a8e1b50-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.400674 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.400662    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/64a8adba-c9b7-4a79-826a-d64a1668f4cf-root\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.400928 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.400688    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.400928 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.400723    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-textfile\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.400928 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.400754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/de118084-296a-4dfb-b5ed-4f869a8e1b50-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.405070 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.404408    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64a8adba-c9b7-4a79-826a-d64a1668f4cf-sys\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.405070 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.405010    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/64a8adba-c9b7-4a79-826a-d64a1668f4cf-root\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.405550 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.405524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64a8adba-c9b7-4a79-826a-d64a1668f4cf-metrics-client-ca\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.405680 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:01:10.405661    2566 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Aug 04 18:01:10.405746 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.405724    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.405802 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:01:10.405734    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de118084-296a-4dfb-b5ed-4f869a8e1b50-kube-state-metrics-tls podName:de118084-296a-4dfb-b5ed-4f869a8e1b50 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:10.905714637 +0000 UTC m=+70.196328803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/de118084-296a-4dfb-b5ed-4f869a8e1b50-kube-state-metrics-tls") pod "kube-state-metrics-6bd9664c4-h46zx" (UID: "de118084-296a-4dfb-b5ed-4f869a8e1b50") : secret "kube-state-metrics-tls" not found
Aug 04 18:01:10.405802 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:01:10.405737    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Aug 04 18:01:10.405986 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.405666    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-wtmp\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.406073 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.406054    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/de118084-296a-4dfb-b5ed-4f869a8e1b50-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.406181 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:01:10.406166    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-tls podName:64a8adba-c9b7-4a79-826a-d64a1668f4cf nodeName:}" failed. No retries permitted until 2026-08-04 18:01:10.906146505 +0000 UTC m=+70.196760658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-tls") pod "node-exporter-k5tqn" (UID: "64a8adba-c9b7-4a79-826a-d64a1668f4cf") : secret "node-exporter-tls" not found
Aug 04 18:01:10.406437 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.406418    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-textfile\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.406852 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.406829    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/de118084-296a-4dfb-b5ed-4f869a8e1b50-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.406960 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.406940    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de118084-296a-4dfb-b5ed-4f869a8e1b50-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.412675 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.412653    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.413236 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.413212    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de118084-296a-4dfb-b5ed-4f869a8e1b50-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.419317 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.419287    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98dbz\" (UniqueName: \"kubernetes.io/projected/64a8adba-c9b7-4a79-826a-d64a1668f4cf-kube-api-access-98dbz\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.422954 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.422929    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5cq8\" (UniqueName: \"kubernetes.io/projected/de118084-296a-4dfb-b5ed-4f869a8e1b50-kube-api-access-h5cq8\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.808125 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.808082    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0f3e4e0-1cb2-4782-a160-b2d4c530fc48-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-kh899\" (UID: \"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"
Aug 04 18:01:10.811120 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.811089    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0f3e4e0-1cb2-4782-a160-b2d4c530fc48-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-kh899\" (UID: \"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"
Aug 04 18:01:10.908665 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.908628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de118084-296a-4dfb-b5ed-4f869a8e1b50-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:10.908846 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.908684    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-tls\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.911545 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.911494    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/64a8adba-c9b7-4a79-826a-d64a1668f4cf-node-exporter-tls\") pod \"node-exporter-k5tqn\" (UID: \"64a8adba-c9b7-4a79-826a-d64a1668f4cf\") " pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:10.911722 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:10.911702    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de118084-296a-4dfb-b5ed-4f869a8e1b50-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-h46zx\" (UID: \"de118084-296a-4dfb-b5ed-4f869a8e1b50\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:11.072270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.072190    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"
Aug 04 18:01:11.094850 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.094822    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"
Aug 04 18:01:11.102003 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.101975    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k5tqn"
Aug 04 18:01:11.138881 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:01:11.138845    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a8adba_c9b7_4a79_826a_d64a1668f4cf.slice/crio-647b8777f1d86c13ff0466882944965aecfb337f22fb69603fd8c7aec2b0b6f5 WatchSource:0}: Error finding container 647b8777f1d86c13ff0466882944965aecfb337f22fb69603fd8c7aec2b0b6f5: Status 404 returned error can't find the container with id 647b8777f1d86c13ff0466882944965aecfb337f22fb69603fd8c7aec2b0b6f5
Aug 04 18:01:11.276616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.276541    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:01:11.290615 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.290591    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.295310 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.294220    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Aug 04 18:01:11.295310 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.294451    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Aug 04 18:01:11.295310 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.294474    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Aug 04 18:01:11.295310 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.294587    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Aug 04 18:01:11.295310 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.294638    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Aug 04 18:01:11.295310 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.294896    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Aug 04 18:01:11.295310 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.294946    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Aug 04 18:01:11.295310 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.295098    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4q9wp\""
Aug 04 18:01:11.295310 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.295138    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Aug 04 18:01:11.295310 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.295272    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Aug 04 18:01:11.306619 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.306592    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:01:11.318672 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.318640    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899"]
Aug 04 18:01:11.335299 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.335221    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx"]
Aug 04 18:01:11.352980 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.352863    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57585d585c-ts8p2"]
Aug 04 18:01:11.401635 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.401608    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xk4xc"
Aug 04 18:01:11.413095 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.413058    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.413257 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.413102    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.413257 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.413144    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp6mg\" (UniqueName: \"kubernetes.io/projected/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-kube-api-access-hp6mg\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.413257 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.413177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-web-config\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.413257 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.413203    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-config-volume\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.413472 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.413276    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.413472 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.413339    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.413472 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.413381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.413472 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.413403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-config-out\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.413607 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.413472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.413607 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.413509    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.413607 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.413527    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.413607 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.413547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.458978 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.458841    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k5tqn" event={"ID":"64a8adba-c9b7-4a79-826a-d64a1668f4cf","Type":"ContainerStarted","Data":"647b8777f1d86c13ff0466882944965aecfb337f22fb69603fd8c7aec2b0b6f5"}
Aug 04 18:01:11.468330 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:01:11.468285    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f3e4e0_1cb2_4782_a160_b2d4c530fc48.slice/crio-2cc5d901aee5ea3f6241f02a8dd139c5f975a6fc087cdd35f105b0e4a09a2e1a WatchSource:0}: Error finding container 2cc5d901aee5ea3f6241f02a8dd139c5f975a6fc087cdd35f105b0e4a09a2e1a: Status 404 returned error can't find the container with id 2cc5d901aee5ea3f6241f02a8dd139c5f975a6fc087cdd35f105b0e4a09a2e1a
Aug 04 18:01:11.469511 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:01:11.469483    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde118084_296a_4dfb_b5ed_4f869a8e1b50.slice/crio-ad03b5545d7f2a135ea180f51bbe9ace45a55eeb7285268c5d047b855fd952c5 WatchSource:0}: Error finding container ad03b5545d7f2a135ea180f51bbe9ace45a55eeb7285268c5d047b855fd952c5: Status 404 returned error can't find the container with id ad03b5545d7f2a135ea180f51bbe9ace45a55eeb7285268c5d047b855fd952c5
Aug 04 18:01:11.471048 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:01:11.471015    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416d258c_7d82_4310_84f3_dc32908b0a4e.slice/crio-5802950bba63f37e63dd1694f72c75b68f498095ea9c85770be411450dd1f449 WatchSource:0}: Error finding container 5802950bba63f37e63dd1694f72c75b68f498095ea9c85770be411450dd1f449: Status 404 returned error can't find the container with id 5802950bba63f37e63dd1694f72c75b68f498095ea9c85770be411450dd1f449
Aug 04 18:01:11.514873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.514842    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.514995 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.514888    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.514995 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.514915    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.514995 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.514943    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.515156 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.515103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.515212 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.515150    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.515212 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.515193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp6mg\" (UniqueName: \"kubernetes.io/projected/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-kube-api-access-hp6mg\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.515388 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.515225    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-web-config\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.515388 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.515252    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-config-volume\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.515388 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.515280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.515388 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.515344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.515592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.515419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.515592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.515441    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-config-out\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.516253 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.515933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.516253 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.515941    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.516444 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.516374    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.518311 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.517963    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.518311 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.518052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.518311 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.518214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-config-volume\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.519312 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.519030    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-web-config\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.519440 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.519404    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.519563 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.519520    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.519738 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.519685    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.520885 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.520764    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.521165 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.521133    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-config-out\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.531302 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.531280    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp6mg\" (UniqueName: \"kubernetes.io/projected/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-kube-api-access-hp6mg\") pod \"alertmanager-main-0\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.609914 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.609759    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:01:11.811188 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:11.811061    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:01:11.814810 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:01:11.814779    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688b3da2_e0c3_4b9d_ac5d_d8410e5d065a.slice/crio-40360cf9c8d4fb839eaa309e0285544875fe4b7d2aeb6c95e95629eb8d469e16 WatchSource:0}: Error finding container 40360cf9c8d4fb839eaa309e0285544875fe4b7d2aeb6c95e95629eb8d469e16: Status 404 returned error can't find the container with id 40360cf9c8d4fb839eaa309e0285544875fe4b7d2aeb6c95e95629eb8d469e16
Aug 04 18:01:12.145849 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.144976    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"]
Aug 04 18:01:12.149410 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.149383    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.152131 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.151895    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Aug 04 18:01:12.152131 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.151920    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Aug 04 18:01:12.152131 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.151927    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Aug 04 18:01:12.152438 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.152419    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Aug 04 18:01:12.152913 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.152657    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-44ud71c6gfu6\""
Aug 04 18:01:12.152913 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.152713    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4fdht\""
Aug 04 18:01:12.152913 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.152743    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Aug 04 18:01:12.164103 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.164077    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"]
Aug 04 18:01:12.223917 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.223791    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.223917 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.223837    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-grpc-tls\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.223917 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.223884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.224092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.223931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqvc6\" (UniqueName: \"kubernetes.io/projected/659adef3-366c-4c1d-b634-650ff98fbb02-kube-api-access-dqvc6\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.224092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.223958    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.224092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.224052    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/659adef3-366c-4c1d-b634-650ff98fbb02-metrics-client-ca\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.224202 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.224109    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-thanos-querier-tls\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.224202 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.224137    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.325041 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.325007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-thanos-querier-tls\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.325041 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.325052    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.325275 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.325094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.325275 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.325133    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-grpc-tls\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.325275 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.325179    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.325275 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.325231    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvc6\" (UniqueName: \"kubernetes.io/projected/659adef3-366c-4c1d-b634-650ff98fbb02-kube-api-access-dqvc6\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.325275 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.325260    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.325540 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.325295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/659adef3-366c-4c1d-b634-650ff98fbb02-metrics-client-ca\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.326090 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.326063    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/659adef3-366c-4c1d-b634-650ff98fbb02-metrics-client-ca\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.329267 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.329215    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.330890 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.329857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.330890 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.329857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-grpc-tls\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.330890 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.330166    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.330890 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.330738    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.332865 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.332823    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/659adef3-366c-4c1d-b634-650ff98fbb02-secret-thanos-querier-tls\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.334415 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.334393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqvc6\" (UniqueName: \"kubernetes.io/projected/659adef3-366c-4c1d-b634-650ff98fbb02-kube-api-access-dqvc6\") pod \"thanos-querier-7588cc9ffd-r4r52\" (UID: \"659adef3-366c-4c1d-b634-650ff98fbb02\") " pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.462235 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.462202    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:12.466502 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.466466    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57585d585c-ts8p2" event={"ID":"416d258c-7d82-4310-84f3-dc32908b0a4e","Type":"ContainerStarted","Data":"ef6fd4cad95e1528c7dda2b2cf3f5d2a8f615fdf13bb91d5c9a069ef5164e701"}
Aug 04 18:01:12.466629 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.466527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57585d585c-ts8p2" event={"ID":"416d258c-7d82-4310-84f3-dc32908b0a4e","Type":"ContainerStarted","Data":"5802950bba63f37e63dd1694f72c75b68f498095ea9c85770be411450dd1f449"}
Aug 04 18:01:12.469787 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.469212    2566 generic.go:356] "Generic (PLEG): container finished" podID="64a8adba-c9b7-4a79-826a-d64a1668f4cf" containerID="9cc94462846c0ef5b2ff4b4b4ddb16b5101d0fbf14c62adee8aface37aa90930" exitCode=0
Aug 04 18:01:12.469787 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.469300    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k5tqn" event={"ID":"64a8adba-c9b7-4a79-826a-d64a1668f4cf","Type":"ContainerDied","Data":"9cc94462846c0ef5b2ff4b4b4ddb16b5101d0fbf14c62adee8aface37aa90930"}
Aug 04 18:01:12.471775 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.471632    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899" event={"ID":"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48","Type":"ContainerStarted","Data":"782518b5c63fe23fee756b456f86c8968a2d93c7730a5b4e7f3efd76298fe168"}
Aug 04 18:01:12.471775 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.471684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899" event={"ID":"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48","Type":"ContainerStarted","Data":"63dcf99cba5c0561b9bf8159310002aee2a08760e7b49f837972a4f38a830fba"}
Aug 04 18:01:12.471775 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.471711    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899" event={"ID":"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48","Type":"ContainerStarted","Data":"2cc5d901aee5ea3f6241f02a8dd139c5f975a6fc087cdd35f105b0e4a09a2e1a"}
Aug 04 18:01:12.474605 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.474568    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx" event={"ID":"de118084-296a-4dfb-b5ed-4f869a8e1b50","Type":"ContainerStarted","Data":"ad03b5545d7f2a135ea180f51bbe9ace45a55eeb7285268c5d047b855fd952c5"}
Aug 04 18:01:12.476897 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.476869    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bd4m4" event={"ID":"dbbadf1e-6e33-48b0-8d9c-437c756cb1eb","Type":"ContainerStarted","Data":"76be4d1445e0fa5bf1b0c4e10431c3e56671b5c230c02a0c85316ef7a092168d"}
Aug 04 18:01:12.478771 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.478723    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a","Type":"ContainerStarted","Data":"40360cf9c8d4fb839eaa309e0285544875fe4b7d2aeb6c95e95629eb8d469e16"}
Aug 04 18:01:12.479722 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.479702    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:01:12.488291 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.488237    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57585d585c-ts8p2" podStartSLOduration=3.488210465 podStartE2EDuration="3.488210465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:12.48772259 +0000 UTC m=+71.778336763" watchObservedRunningTime="2026-08-04 18:01:12.488210465 +0000 UTC m=+71.778824637"
Aug 04 18:01:12.532104 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.526277    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bd4m4" podStartSLOduration=67.50455482 podStartE2EDuration="1m11.526256415s" totalImagesPullingTime="4.021701595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:12.525791894 +0000 UTC m=+71.816406088" watchObservedRunningTime="2026-08-04 18:01:12.526256415 +0000 UTC m=+71.816870586"
Aug 04 18:01:12.648767 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:12.648654    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"]
Aug 04 18:01:13.485199 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:13.485115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k5tqn" event={"ID":"64a8adba-c9b7-4a79-826a-d64a1668f4cf","Type":"ContainerStarted","Data":"5d15e4a1a723552c7e97c30dd7836dbab480758a577babff1a061cde3d3d93fb"}
Aug 04 18:01:14.019147 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:01:14.019118    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659adef3_366c_4c1d_b634_650ff98fbb02.slice/crio-d3d3887decf9b6dbfa9cfa7cb7ff8d7a269d2f800c225b4d9e7c3ff7d921bf48 WatchSource:0}: Error finding container d3d3887decf9b6dbfa9cfa7cb7ff8d7a269d2f800c225b4d9e7c3ff7d921bf48: Status 404 returned error can't find the container with id d3d3887decf9b6dbfa9cfa7cb7ff8d7a269d2f800c225b4d9e7c3ff7d921bf48
Aug 04 18:01:14.096712 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.096640    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:14.096712 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.096715    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:14.102164 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.102137    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:14.490992 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.490954    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52" event={"ID":"659adef3-366c-4c1d-b634-650ff98fbb02","Type":"ContainerStarted","Data":"d3d3887decf9b6dbfa9cfa7cb7ff8d7a269d2f800c225b4d9e7c3ff7d921bf48"}
Aug 04 18:01:14.495818 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.495792    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:14.524860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.524831    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-657674cc-656rm"]
Aug 04 18:01:14.527474 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.527450    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.530557 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.530531    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Aug 04 18:01:14.530797 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.530750    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Aug 04 18:01:14.530911 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.530852    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Aug 04 18:01:14.531301 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.530997    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Aug 04 18:01:14.531301 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.531066    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-gvtsfvjo2mms\""
Aug 04 18:01:14.531301 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.531202    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pdbh7\""
Aug 04 18:01:14.540814 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.540793    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-657674cc-656rm"]
Aug 04 18:01:14.651592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.651449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2664c33b-5edb-4026-adb7-ca112956f898-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.651592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.651513    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2664c33b-5edb-4026-adb7-ca112956f898-secret-metrics-server-tls\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.651592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.651561    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txmx7\" (UniqueName: \"kubernetes.io/projected/2664c33b-5edb-4026-adb7-ca112956f898-kube-api-access-txmx7\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.651592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.651591    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2664c33b-5edb-4026-adb7-ca112956f898-metrics-server-audit-profiles\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.651913 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.651671    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2664c33b-5edb-4026-adb7-ca112956f898-secret-metrics-server-client-certs\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.651913 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.651712    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2664c33b-5edb-4026-adb7-ca112956f898-client-ca-bundle\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.651913 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.651740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2664c33b-5edb-4026-adb7-ca112956f898-audit-log\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.753616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.753310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2664c33b-5edb-4026-adb7-ca112956f898-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.753616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.753397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2664c33b-5edb-4026-adb7-ca112956f898-secret-metrics-server-tls\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.753616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.753440    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txmx7\" (UniqueName: \"kubernetes.io/projected/2664c33b-5edb-4026-adb7-ca112956f898-kube-api-access-txmx7\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.753616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.753469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2664c33b-5edb-4026-adb7-ca112956f898-metrics-server-audit-profiles\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.753616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.753502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2664c33b-5edb-4026-adb7-ca112956f898-secret-metrics-server-client-certs\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.753616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.753542    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2664c33b-5edb-4026-adb7-ca112956f898-client-ca-bundle\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.753616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.753569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2664c33b-5edb-4026-adb7-ca112956f898-audit-log\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.754186 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.754035    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2664c33b-5edb-4026-adb7-ca112956f898-audit-log\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.754707 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.754649    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2664c33b-5edb-4026-adb7-ca112956f898-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.755142 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.755080    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2664c33b-5edb-4026-adb7-ca112956f898-metrics-server-audit-profiles\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.756882 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.756737    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2664c33b-5edb-4026-adb7-ca112956f898-client-ca-bundle\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.756882 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.756823    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2664c33b-5edb-4026-adb7-ca112956f898-secret-metrics-server-client-certs\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.760480 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.760333    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2664c33b-5edb-4026-adb7-ca112956f898-secret-metrics-server-tls\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.764855 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.764805    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txmx7\" (UniqueName: \"kubernetes.io/projected/2664c33b-5edb-4026-adb7-ca112956f898-kube-api-access-txmx7\") pod \"metrics-server-657674cc-656rm\" (UID: \"2664c33b-5edb-4026-adb7-ca112956f898\") " pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.840195 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.840162    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:14.914370 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.914326    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-4zk4t"]
Aug 04 18:01:14.917045 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.917026    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-4zk4t"
Aug 04 18:01:14.921561 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.921534    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tgvvf\""
Aug 04 18:01:14.921813 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.921794    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Aug 04 18:01:14.927661 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:14.927466    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-4zk4t"]
Aug 04 18:01:15.048275 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.047914    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-657674cc-656rm"]
Aug 04 18:01:15.051129 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:01:15.051106    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2664c33b_5edb_4026_adb7_ca112956f898.slice/crio-3159f9f713190d5e2c54b69e2294a75278e2aea2faac471d6a6c917e0afc4cfb WatchSource:0}: Error finding container 3159f9f713190d5e2c54b69e2294a75278e2aea2faac471d6a6c917e0afc4cfb: Status 404 returned error can't find the container with id 3159f9f713190d5e2c54b69e2294a75278e2aea2faac471d6a6c917e0afc4cfb
Aug 04 18:01:15.057511 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.057484    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj7ss\" (UniqueName: \"kubernetes.io/projected/ec4c43a8-531d-4176-a3a7-51541eb13a3e-kube-api-access-hj7ss\") pod \"monitoring-plugin-6dffbbd67-4zk4t\" (UID: \"ec4c43a8-531d-4176-a3a7-51541eb13a3e\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-4zk4t"
Aug 04 18:01:15.057604 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.057534    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec4c43a8-531d-4176-a3a7-51541eb13a3e-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-4zk4t\" (UID: \"ec4c43a8-531d-4176-a3a7-51541eb13a3e\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-4zk4t"
Aug 04 18:01:15.158568 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.158475    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj7ss\" (UniqueName: \"kubernetes.io/projected/ec4c43a8-531d-4176-a3a7-51541eb13a3e-kube-api-access-hj7ss\") pod \"monitoring-plugin-6dffbbd67-4zk4t\" (UID: \"ec4c43a8-531d-4176-a3a7-51541eb13a3e\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-4zk4t"
Aug 04 18:01:15.158568 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.158541    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec4c43a8-531d-4176-a3a7-51541eb13a3e-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-4zk4t\" (UID: \"ec4c43a8-531d-4176-a3a7-51541eb13a3e\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-4zk4t"
Aug 04 18:01:15.161559 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.161526    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec4c43a8-531d-4176-a3a7-51541eb13a3e-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-4zk4t\" (UID: \"ec4c43a8-531d-4176-a3a7-51541eb13a3e\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-4zk4t"
Aug 04 18:01:15.167753 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.167724    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj7ss\" (UniqueName: \"kubernetes.io/projected/ec4c43a8-531d-4176-a3a7-51541eb13a3e-kube-api-access-hj7ss\") pod \"monitoring-plugin-6dffbbd67-4zk4t\" (UID: \"ec4c43a8-531d-4176-a3a7-51541eb13a3e\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-4zk4t"
Aug 04 18:01:15.234968 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.234928    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-4zk4t"
Aug 04 18:01:15.409941 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.409862    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-4zk4t"]
Aug 04 18:01:15.410918 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.410897    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-84f56b86-95mmb"]
Aug 04 18:01:15.412688 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:01:15.412663    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4c43a8_531d_4176_a3a7_51541eb13a3e.slice/crio-fd3109551a207a738816954af766dc504d6f8564366e7765517e42b358bc27d4 WatchSource:0}: Error finding container fd3109551a207a738816954af766dc504d6f8564366e7765517e42b358bc27d4: Status 404 returned error can't find the container with id fd3109551a207a738816954af766dc504d6f8564366e7765517e42b358bc27d4
Aug 04 18:01:15.413585 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.413567    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.416382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.416339    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Aug 04 18:01:15.416494 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.416477    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Aug 04 18:01:15.416557 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.416507    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Aug 04 18:01:15.416557 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.416520    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-28rjp\""
Aug 04 18:01:15.416663 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.416600    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Aug 04 18:01:15.416911 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.416879    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Aug 04 18:01:15.421006 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.420979    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Aug 04 18:01:15.426688 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.426664    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-84f56b86-95mmb"]
Aug 04 18:01:15.498150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.498116    2566 generic.go:356] "Generic (PLEG): container finished" podID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerID="3c3b0047649f5d3d53c4b08d73a0855948512f65e65d6185564553c2d2728aac" exitCode=0
Aug 04 18:01:15.498590 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.498316    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a","Type":"ContainerDied","Data":"3c3b0047649f5d3d53c4b08d73a0855948512f65e65d6185564553c2d2728aac"}
Aug 04 18:01:15.499732 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.499707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-657674cc-656rm" event={"ID":"2664c33b-5edb-4026-adb7-ca112956f898","Type":"ContainerStarted","Data":"3159f9f713190d5e2c54b69e2294a75278e2aea2faac471d6a6c917e0afc4cfb"}
Aug 04 18:01:15.501563 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.501539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t46pw" event={"ID":"6cbc3f49-e03c-425d-9da2-71eb15915d42","Type":"ContainerStarted","Data":"21eda6fc3315f56a1b5c6f287d236cacf6b60deb29229d172c875ada6dd0f1df"}
Aug 04 18:01:15.504175 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.504148    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k5tqn" event={"ID":"64a8adba-c9b7-4a79-826a-d64a1668f4cf","Type":"ContainerStarted","Data":"c1b9769d98bad6f81f193bf7d6ca186ee7982a46e72a3890fea2802e5c535cd2"}
Aug 04 18:01:15.506606 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.506584    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899" event={"ID":"d0f3e4e0-1cb2-4782-a160-b2d4c530fc48","Type":"ContainerStarted","Data":"5f49dcdc71931b565c96e7f3612244847eacc0af07313d91091b60d8b80ee0eb"}
Aug 04 18:01:15.509117 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.509085    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx" event={"ID":"de118084-296a-4dfb-b5ed-4f869a8e1b50","Type":"ContainerStarted","Data":"3bbf8a11a554a3f3bdd6b71e9a011992c849f06df483261f12e88771c23b19ca"}
Aug 04 18:01:15.509214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.509137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx" event={"ID":"de118084-296a-4dfb-b5ed-4f869a8e1b50","Type":"ContainerStarted","Data":"24d28555e4b48eae4b91489d2101b42c37a8a0ea44f8f8dff4da99544a261743"}
Aug 04 18:01:15.509214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.509167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx" event={"ID":"de118084-296a-4dfb-b5ed-4f869a8e1b50","Type":"ContainerStarted","Data":"5ed16b75a5d9c45ef26d997e5221f847e8c2381308dde340a01622d674d35906"}
Aug 04 18:01:15.510515 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.510487    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-4zk4t" event={"ID":"ec4c43a8-531d-4176-a3a7-51541eb13a3e","Type":"ContainerStarted","Data":"fd3109551a207a738816954af766dc504d6f8564366e7765517e42b358bc27d4"}
Aug 04 18:01:15.544137 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.544073    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-kh899" podStartSLOduration=1.983020222 podStartE2EDuration="5.544052791s" totalImagesPullingTime="3.561032569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:15.542692317 +0000 UTC m=+74.833306488" watchObservedRunningTime="2026-08-04 18:01:15.544052791 +0000 UTC m=+74.834666963"
Aug 04 18:01:15.562614 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.562580    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f36fbfde-f663-49b3-b8c1-6160b8243eca-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.562773 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.562625    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f36fbfde-f663-49b3-b8c1-6160b8243eca-federate-client-tls\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.562773 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.562684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f36fbfde-f663-49b3-b8c1-6160b8243eca-secret-telemeter-client\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.562773 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.562706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f36fbfde-f663-49b3-b8c1-6160b8243eca-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.562891 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.562831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f36fbfde-f663-49b3-b8c1-6160b8243eca-metrics-client-ca\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.563169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.563150    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f36fbfde-f663-49b3-b8c1-6160b8243eca-serving-certs-ca-bundle\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.563361 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.563310    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f36fbfde-f663-49b3-b8c1-6160b8243eca-telemeter-client-tls\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.563491 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.563430    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwcj5\" (UniqueName: \"kubernetes.io/projected/f36fbfde-f663-49b3-b8c1-6160b8243eca-kube-api-access-bwcj5\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.567847 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.564735    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-h46zx" podStartSLOduration=2.448442041 podStartE2EDuration="5.564716556s" totalImagesPullingTime="3.116274515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:15.56094987 +0000 UTC m=+74.851564085" watchObservedRunningTime="2026-08-04 18:01:15.564716556 +0000 UTC m=+74.855330729"
Aug 04 18:01:15.586802 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.586751    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t46pw" podStartSLOduration=67.364724433 podStartE2EDuration="1m14.58673833s" totalImagesPullingTime="7.222013897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:15.586050798 +0000 UTC m=+74.876664993" watchObservedRunningTime="2026-08-04 18:01:15.58673833 +0000 UTC m=+74.877352501"
Aug 04 18:01:15.608501 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.608441    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k5tqn" podStartSLOduration=4.447631756 podStartE2EDuration="5.608419354s" totalImagesPullingTime="1.074807629s" totalInitContainerRuntime="85.979969ms" isStatefulPod=true podCreationTimestamp="2026-08-04 18:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:15.607406741 +0000 UTC m=+74.898020931" watchObservedRunningTime="2026-08-04 18:01:15.608419354 +0000 UTC m=+74.899033526"
Aug 04 18:01:15.664341 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.664231    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f36fbfde-f663-49b3-b8c1-6160b8243eca-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.664341 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.664283    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f36fbfde-f663-49b3-b8c1-6160b8243eca-federate-client-tls\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.664571 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.664347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f36fbfde-f663-49b3-b8c1-6160b8243eca-secret-telemeter-client\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.664571 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.664388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f36fbfde-f663-49b3-b8c1-6160b8243eca-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.664571 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.664508    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f36fbfde-f663-49b3-b8c1-6160b8243eca-metrics-client-ca\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.664571 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.664553    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f36fbfde-f663-49b3-b8c1-6160b8243eca-serving-certs-ca-bundle\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.664759 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.664647    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f36fbfde-f663-49b3-b8c1-6160b8243eca-telemeter-client-tls\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.664759 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.664728    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwcj5\" (UniqueName: \"kubernetes.io/projected/f36fbfde-f663-49b3-b8c1-6160b8243eca-kube-api-access-bwcj5\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.665673 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.665478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f36fbfde-f663-49b3-b8c1-6160b8243eca-metrics-client-ca\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.665673 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.665667    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f36fbfde-f663-49b3-b8c1-6160b8243eca-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.666149 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.666099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f36fbfde-f663-49b3-b8c1-6160b8243eca-serving-certs-ca-bundle\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.667792 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.667748    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f36fbfde-f663-49b3-b8c1-6160b8243eca-federate-client-tls\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.667792 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.667786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f36fbfde-f663-49b3-b8c1-6160b8243eca-telemeter-client-tls\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.667947 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.667833    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f36fbfde-f663-49b3-b8c1-6160b8243eca-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.667947 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.667846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f36fbfde-f663-49b3-b8c1-6160b8243eca-secret-telemeter-client\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.674208 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.674184    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwcj5\" (UniqueName: \"kubernetes.io/projected/f36fbfde-f663-49b3-b8c1-6160b8243eca-kube-api-access-bwcj5\") pod \"telemeter-client-84f56b86-95mmb\" (UID: \"f36fbfde-f663-49b3-b8c1-6160b8243eca\") " pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.724590 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.724558    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-84f56b86-95mmb"
Aug 04 18:01:15.885192 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:15.885158    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-84f56b86-95mmb"]
Aug 04 18:01:15.888410 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:01:15.888342    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36fbfde_f663_49b3_b8c1_6160b8243eca.slice/crio-facdb1cc2571976fff02d43f0c18a1cc790f1fb91eea813cc981e935e2b0be90 WatchSource:0}: Error finding container facdb1cc2571976fff02d43f0c18a1cc790f1fb91eea813cc981e935e2b0be90: Status 404 returned error can't find the container with id facdb1cc2571976fff02d43f0c18a1cc790f1fb91eea813cc981e935e2b0be90
Aug 04 18:01:16.474645 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.474607    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:01:16.480999 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.478691    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.481755 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.481639    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Aug 04 18:01:16.482759 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.482138    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e4kbmrl6ugl4\""
Aug 04 18:01:16.482759 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.482266    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Aug 04 18:01:16.482759 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.482367    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Aug 04 18:01:16.482759 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.482593    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Aug 04 18:01:16.482759 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.482736    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Aug 04 18:01:16.483075 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.482784    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Aug 04 18:01:16.483075 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.483056    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Aug 04 18:01:16.483423 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.483253    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Aug 04 18:01:16.484139 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.483994    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zj94x\""
Aug 04 18:01:16.484139 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.484111    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Aug 04 18:01:16.484283 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.484146    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Aug 04 18:01:16.484508 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.484480    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Aug 04 18:01:16.484877 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.484520    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Aug 04 18:01:16.490908 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.490889    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Aug 04 18:01:16.492318 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.492286    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Aug 04 18:01:16.498339 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.498250    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:01:16.519693 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.519648    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84f56b86-95mmb" event={"ID":"f36fbfde-f663-49b3-b8c1-6160b8243eca","Type":"ContainerStarted","Data":"facdb1cc2571976fff02d43f0c18a1cc790f1fb91eea813cc981e935e2b0be90"}
Aug 04 18:01:16.575663 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.575609    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.575663 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.575660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.575904 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.575688    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.575904 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.575720    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.575904 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.575766    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.575904 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.575792    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.575904 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.575815    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.575904 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.575864    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.576207 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.575907    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-config\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.576207 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.575930    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.576207 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.575963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-web-config\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.576207 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.576000    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.576207 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.576041    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f63bfef0-7395-4857-9f3b-a59161a9703b-config-out\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.576207 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.576064    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.576207 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.576095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.576207 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.576137    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.576207 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.576162    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.576207 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.576183    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.576702 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.576214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfsc4\" (UniqueName: \"kubernetes.io/projected/f63bfef0-7395-4857-9f3b-a59161a9703b-kube-api-access-wfsc4\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.576702 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.576254    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f63bfef0-7395-4857-9f3b-a59161a9703b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.677595    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-config\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.677643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.677701    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-web-config\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.677751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.677796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f63bfef0-7395-4857-9f3b-a59161a9703b-config-out\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.677819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.677848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.677907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.677931    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.677954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.677986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfsc4\" (UniqueName: \"kubernetes.io/projected/f63bfef0-7395-4857-9f3b-a59161a9703b-kube-api-access-wfsc4\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.678035    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f63bfef0-7395-4857-9f3b-a59161a9703b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.678124    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.678160    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.678188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.678243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.679042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.678326    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.680618 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.680548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.680764 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.680728    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.680924 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.680879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.682656 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.681152    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.682656 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.681919    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.682656 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.682432    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.682656 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.682564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.683519 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.683319    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.683620 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.683594    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.688387 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.687686    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.688387 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.687991    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.688387 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.688376    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f63bfef0-7395-4857-9f3b-a59161a9703b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.696930 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.696884    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-config\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.697427 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.697151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-web-config\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.702727 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.701307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.702727 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.701555    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f63bfef0-7395-4857-9f3b-a59161a9703b-config-out\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.702727 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.702070    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.702727 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.702104    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.702727 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.702436    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.702727 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.702678    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.703091 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.702944    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.703750 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.703725    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.705432 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.705324    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfsc4\" (UniqueName: \"kubernetes.io/projected/f63bfef0-7395-4857-9f3b-a59161a9703b-kube-api-access-wfsc4\") pod \"prometheus-k8s-0\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:16.799991 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:16.799680    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:17.231397 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:17.231279    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:01:17.526419 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:17.526326    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-657674cc-656rm" event={"ID":"2664c33b-5edb-4026-adb7-ca112956f898","Type":"ContainerStarted","Data":"82c6f43cb317776226fcd1f5ff8ff7bce94d655cb7e74e519121e3805db7e168"}
Aug 04 18:01:17.546196 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:17.546129    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-657674cc-656rm" podStartSLOduration=1.553810569 podStartE2EDuration="3.546109045s" totalImagesPullingTime="1.992298476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:17.545939393 +0000 UTC m=+76.836553566" watchObservedRunningTime="2026-08-04 18:01:17.546109045 +0000 UTC m=+76.836723229"
Aug 04 18:01:17.614154 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:01:17.614004    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63bfef0_7395_4857_9f3b_a59161a9703b.slice/crio-7a748fe1f9abbc96aaa93c67052df89e1da41f97485de638f3369c4722549c6f WatchSource:0}: Error finding container 7a748fe1f9abbc96aaa93c67052df89e1da41f97485de638f3369c4722549c6f: Status 404 returned error can't find the container with id 7a748fe1f9abbc96aaa93c67052df89e1da41f97485de638f3369c4722549c6f
Aug 04 18:01:17.775892 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:17.775285    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57585d585c-ts8p2"]
Aug 04 18:01:18.532266 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:18.532227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f63bfef0-7395-4857-9f3b-a59161a9703b","Type":"ContainerStarted","Data":"7a748fe1f9abbc96aaa93c67052df89e1da41f97485de638f3369c4722549c6f"}
Aug 04 18:01:19.538535 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.538494    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52" event={"ID":"659adef3-366c-4c1d-b634-650ff98fbb02","Type":"ContainerStarted","Data":"329ae29ed6fb6abc384c2eb89a066c11cccececd610b6bf5b78dddb6ce8fd2c5"}
Aug 04 18:01:19.538920 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.538585    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52" event={"ID":"659adef3-366c-4c1d-b634-650ff98fbb02","Type":"ContainerStarted","Data":"26fbe82b45df2e826870f5b6c4146cf0815b6f05c5f57394031f9a0346e27a65"}
Aug 04 18:01:19.538920 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.538628    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52" event={"ID":"659adef3-366c-4c1d-b634-650ff98fbb02","Type":"ContainerStarted","Data":"4e2c70572ab33a29cfd778ad23f9160b225e2fc73e9e4a8d1aa798d5d7e89bb3"}
Aug 04 18:01:19.540006 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.539985    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-4zk4t" event={"ID":"ec4c43a8-531d-4176-a3a7-51541eb13a3e","Type":"ContainerStarted","Data":"b42623016c7499207bbfefc5f6cf981bcf2055734ca68c7fd83d00b3444d47ec"}
Aug 04 18:01:19.540162 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.540127    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-4zk4t"
Aug 04 18:01:19.543931 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.543729    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a","Type":"ContainerStarted","Data":"d1db49503a42e1d3111a5e27c06ef6ef438e2ef3433347aeff062eca9b5d59ce"}
Aug 04 18:01:19.543931 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.543802    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a","Type":"ContainerStarted","Data":"13bb477364f0f8002ecfc4a3a03a51eb0ded223878150d7779cf7dbe8d7e163d"}
Aug 04 18:01:19.543931 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.543842    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a","Type":"ContainerStarted","Data":"d319d9d147e73b6d7ca3a6d8f0ede45c897f15b10101e923a2274d113300b03c"}
Aug 04 18:01:19.543931 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.543867    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a","Type":"ContainerStarted","Data":"cc17ea317cca9b5737d581ddb96e08e5e22ca19798d21753b8cf2d49156019d8"}
Aug 04 18:01:19.543931 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.543891    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a","Type":"ContainerStarted","Data":"c1814e365acd691cf56fcb58666f519d8824188685c0c1c88ca18145d7a05882"}
Aug 04 18:01:19.545749 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.545727    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-4zk4t"
Aug 04 18:01:19.546402 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.546382    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84f56b86-95mmb" event={"ID":"f36fbfde-f663-49b3-b8c1-6160b8243eca","Type":"ContainerStarted","Data":"d39a3168c36a47e3d82f6bb1fa6bd7671e8ef99c3bafdd52e71e1d25cc69c3e6"}
Aug 04 18:01:19.546495 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.546423    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84f56b86-95mmb" event={"ID":"f36fbfde-f663-49b3-b8c1-6160b8243eca","Type":"ContainerStarted","Data":"7579b55e0d57373efe78eb31c96c30e5012dbbb9a24416647eaedc902a18192b"}
Aug 04 18:01:19.546495 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.546453    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84f56b86-95mmb" event={"ID":"f36fbfde-f663-49b3-b8c1-6160b8243eca","Type":"ContainerStarted","Data":"2c6d5b2b68934df7293de257b2b1d8e73302a03f2d95e8407dcea5644320b77e"}
Aug 04 18:01:19.547871 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.547849    2566 generic.go:356] "Generic (PLEG): container finished" podID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerID="a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885" exitCode=0
Aug 04 18:01:19.547957 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.547885    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f63bfef0-7395-4857-9f3b-a59161a9703b","Type":"ContainerDied","Data":"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885"}
Aug 04 18:01:19.558109 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.558067    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-4zk4t" podStartSLOduration=2.287167703 podStartE2EDuration="5.558053843s" totalImagesPullingTime="3.27088614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:19.557536768 +0000 UTC m=+78.848150940" watchObservedRunningTime="2026-08-04 18:01:19.558053843 +0000 UTC m=+78.848668014"
Aug 04 18:01:19.583203 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:19.583153    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-84f56b86-95mmb" podStartSLOduration=1.786496937 podStartE2EDuration="4.583135841s" totalImagesPullingTime="2.796638904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:19.58216778 +0000 UTC m=+78.872781976" watchObservedRunningTime="2026-08-04 18:01:19.583135841 +0000 UTC m=+78.873750011"
Aug 04 18:01:20.266975 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:20.266890    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:20.556690 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:20.556597    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a","Type":"ContainerStarted","Data":"2d3c698f80f9ea3d860b646f371c3c5e65f41ed1c8ce923e2f1989a55c4a7945"}
Aug 04 18:01:20.586209 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:20.586150    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.488885749 podStartE2EDuration="9.586130894s" totalImagesPullingTime="5.071808795s" totalInitContainerRuntime="25.43635ms" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:20.582856827 +0000 UTC m=+79.873471000" watchObservedRunningTime="2026-08-04 18:01:20.586130894 +0000 UTC m=+79.876745067"
Aug 04 18:01:22.570957 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:22.570913    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52" event={"ID":"659adef3-366c-4c1d-b634-650ff98fbb02","Type":"ContainerStarted","Data":"d46e8d2df5655c35591cbf3b3bb82c61c8c703f4fae1af0968f05a4277b1090f"}
Aug 04 18:01:22.571419 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:22.571000    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52" event={"ID":"659adef3-366c-4c1d-b634-650ff98fbb02","Type":"ContainerStarted","Data":"f751d3d3b0e5e3a4f9ee3274be5a6d5d25d8492f1c940ed643c094d17d782acc"}
Aug 04 18:01:22.571419 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:22.571038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52" event={"ID":"659adef3-366c-4c1d-b634-650ff98fbb02","Type":"ContainerStarted","Data":"dd9a93dc54e368100dc3186bb684223d00dce72c0906b6748b21cf4ad868a2ca"}
Aug 04 18:01:22.571419 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:22.571077    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:22.601781 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:22.601722    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52" podStartSLOduration=3.257138813 podStartE2EDuration="10.601705302s" totalImagesPullingTime="7.344566489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:22.599098096 +0000 UTC m=+81.889712309" watchObservedRunningTime="2026-08-04 18:01:22.601705302 +0000 UTC m=+81.892319471"
Aug 04 18:01:23.579835 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:23.579805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f63bfef0-7395-4857-9f3b-a59161a9703b","Type":"ContainerStarted","Data":"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c"}
Aug 04 18:01:23.580148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:23.579912    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f63bfef0-7395-4857-9f3b-a59161a9703b","Type":"ContainerStarted","Data":"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e"}
Aug 04 18:01:23.580148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:23.579945    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f63bfef0-7395-4857-9f3b-a59161a9703b","Type":"ContainerStarted","Data":"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83"}
Aug 04 18:01:23.580148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:23.579979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f63bfef0-7395-4857-9f3b-a59161a9703b","Type":"ContainerStarted","Data":"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0"}
Aug 04 18:01:23.587051 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:23.587028    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7588cc9ffd-r4r52"
Aug 04 18:01:24.589208 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:24.589172    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f63bfef0-7395-4857-9f3b-a59161a9703b","Type":"ContainerStarted","Data":"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131"}
Aug 04 18:01:24.589685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:24.589245    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f63bfef0-7395-4857-9f3b-a59161a9703b","Type":"ContainerStarted","Data":"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460"}
Aug 04 18:01:24.622983 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:24.622923    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.897577919 podStartE2EDuration="8.62290071s" totalImagesPullingTime="3.697568128s" totalInitContainerRuntime="27.754663ms" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:24.621056944 +0000 UTC m=+83.911671137" watchObservedRunningTime="2026-08-04 18:01:24.62290071 +0000 UTC m=+83.913514881"
Aug 04 18:01:26.800432 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:26.800395    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:01:31.698300 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:31.698259    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68bfcb7466-h97jl"]
Aug 04 18:01:34.840728 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:34.840677    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:34.841111 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:34.840779    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:42.808287 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:42.808218    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57585d585c-ts8p2" podUID="416d258c-7d82-4310-84f3-dc32908b0a4e" containerName="console" containerID="cri-o://ef6fd4cad95e1528c7dda2b2cf3f5d2a8f615fdf13bb91d5c9a069ef5164e701" gracePeriod=15
Aug 04 18:01:43.093892 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.093817    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:43.136909 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.136879    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/416d258c-7d82-4310-84f3-dc32908b0a4e-console-serving-cert\") pod \"416d258c-7d82-4310-84f3-dc32908b0a4e\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") "
Aug 04 18:01:43.137065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.136929    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-oauth-serving-cert\") pod \"416d258c-7d82-4310-84f3-dc32908b0a4e\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") "
Aug 04 18:01:43.137065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.136958    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-console-config\") pod \"416d258c-7d82-4310-84f3-dc32908b0a4e\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") "
Aug 04 18:01:43.137156 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.137071    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/416d258c-7d82-4310-84f3-dc32908b0a4e-console-oauth-config\") pod \"416d258c-7d82-4310-84f3-dc32908b0a4e\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") "
Aug 04 18:01:43.137156 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.137138    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvfkm\" (UniqueName: \"kubernetes.io/projected/416d258c-7d82-4310-84f3-dc32908b0a4e-kube-api-access-hvfkm\") pod \"416d258c-7d82-4310-84f3-dc32908b0a4e\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") "
Aug 04 18:01:43.137257 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.137198    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-trusted-ca-bundle\") pod \"416d258c-7d82-4310-84f3-dc32908b0a4e\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") "
Aug 04 18:01:43.137257 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.137236    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-service-ca\") pod \"416d258c-7d82-4310-84f3-dc32908b0a4e\" (UID: \"416d258c-7d82-4310-84f3-dc32908b0a4e\") "
Aug 04 18:01:43.137381 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.137308    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "416d258c-7d82-4310-84f3-dc32908b0a4e" (UID: "416d258c-7d82-4310-84f3-dc32908b0a4e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:43.137446 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.137318    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-console-config" (OuterVolumeSpecName: "console-config") pod "416d258c-7d82-4310-84f3-dc32908b0a4e" (UID: "416d258c-7d82-4310-84f3-dc32908b0a4e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:43.137598 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.137578    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-oauth-serving-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:01:43.137681 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.137610    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-console-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:01:43.137681 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.137652    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-service-ca" (OuterVolumeSpecName: "service-ca") pod "416d258c-7d82-4310-84f3-dc32908b0a4e" (UID: "416d258c-7d82-4310-84f3-dc32908b0a4e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:43.137785 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.137700    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "416d258c-7d82-4310-84f3-dc32908b0a4e" (UID: "416d258c-7d82-4310-84f3-dc32908b0a4e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:43.139363 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.139324    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416d258c-7d82-4310-84f3-dc32908b0a4e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "416d258c-7d82-4310-84f3-dc32908b0a4e" (UID: "416d258c-7d82-4310-84f3-dc32908b0a4e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:01:43.139450 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.139334    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416d258c-7d82-4310-84f3-dc32908b0a4e-kube-api-access-hvfkm" (OuterVolumeSpecName: "kube-api-access-hvfkm") pod "416d258c-7d82-4310-84f3-dc32908b0a4e" (UID: "416d258c-7d82-4310-84f3-dc32908b0a4e"). InnerVolumeSpecName "kube-api-access-hvfkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:01:43.139450 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.139407    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416d258c-7d82-4310-84f3-dc32908b0a4e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "416d258c-7d82-4310-84f3-dc32908b0a4e" (UID: "416d258c-7d82-4310-84f3-dc32908b0a4e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:01:43.238496 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.238454    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvfkm\" (UniqueName: \"kubernetes.io/projected/416d258c-7d82-4310-84f3-dc32908b0a4e-kube-api-access-hvfkm\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:01:43.238496 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.238486    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-trusted-ca-bundle\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:01:43.238496 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.238498    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/416d258c-7d82-4310-84f3-dc32908b0a4e-service-ca\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:01:43.238496 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.238506    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/416d258c-7d82-4310-84f3-dc32908b0a4e-console-serving-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:01:43.238843 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.238515    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/416d258c-7d82-4310-84f3-dc32908b0a4e-console-oauth-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:01:43.676783 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.676748    2566 generic.go:356] "Generic (PLEG): container finished" podID="416d258c-7d82-4310-84f3-dc32908b0a4e" containerID="ef6fd4cad95e1528c7dda2b2cf3f5d2a8f615fdf13bb91d5c9a069ef5164e701" exitCode=0
Aug 04 18:01:43.676978 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.676793    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57585d585c-ts8p2"
Aug 04 18:01:43.676978 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.676834    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57585d585c-ts8p2" event={"ID":"416d258c-7d82-4310-84f3-dc32908b0a4e","Type":"ContainerDied","Data":"ef6fd4cad95e1528c7dda2b2cf3f5d2a8f615fdf13bb91d5c9a069ef5164e701"}
Aug 04 18:01:43.676978 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.676892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57585d585c-ts8p2" event={"ID":"416d258c-7d82-4310-84f3-dc32908b0a4e","Type":"ContainerDied","Data":"5802950bba63f37e63dd1694f72c75b68f498095ea9c85770be411450dd1f449"}
Aug 04 18:01:43.676978 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.676932    2566 scope.go:117] "RemoveContainer" containerID="ef6fd4cad95e1528c7dda2b2cf3f5d2a8f615fdf13bb91d5c9a069ef5164e701"
Aug 04 18:01:43.685094 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.685077    2566 scope.go:117] "RemoveContainer" containerID="ef6fd4cad95e1528c7dda2b2cf3f5d2a8f615fdf13bb91d5c9a069ef5164e701"
Aug 04 18:01:43.685385 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:01:43.685346    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6fd4cad95e1528c7dda2b2cf3f5d2a8f615fdf13bb91d5c9a069ef5164e701\": container with ID starting with ef6fd4cad95e1528c7dda2b2cf3f5d2a8f615fdf13bb91d5c9a069ef5164e701 not found: ID does not exist" containerID="ef6fd4cad95e1528c7dda2b2cf3f5d2a8f615fdf13bb91d5c9a069ef5164e701"
Aug 04 18:01:43.685497 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.685388    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6fd4cad95e1528c7dda2b2cf3f5d2a8f615fdf13bb91d5c9a069ef5164e701"} err="failed to get container status \"ef6fd4cad95e1528c7dda2b2cf3f5d2a8f615fdf13bb91d5c9a069ef5164e701\": rpc error: code = NotFound desc = could not find container \"ef6fd4cad95e1528c7dda2b2cf3f5d2a8f615fdf13bb91d5c9a069ef5164e701\": container with ID starting with ef6fd4cad95e1528c7dda2b2cf3f5d2a8f615fdf13bb91d5c9a069ef5164e701 not found: ID does not exist"
Aug 04 18:01:43.696827 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.696800    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57585d585c-ts8p2"]
Aug 04 18:01:43.701040 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:43.701019    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57585d585c-ts8p2"]
Aug 04 18:01:44.493843 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:44.493816    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bd4m4"
Aug 04 18:01:45.193902 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:45.193871    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="416d258c-7d82-4310-84f3-dc32908b0a4e" path="/var/lib/kubelet/pods/416d258c-7d82-4310-84f3-dc32908b0a4e/volumes"
Aug 04 18:01:54.846272 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:54.846242    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:54.849930 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:54.849903    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-657674cc-656rm"
Aug 04 18:01:56.717375 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:56.717288    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68bfcb7466-h97jl" podUID="db17423e-c693-4630-a50d-d8b5567f2051" containerName="console" containerID="cri-o://1acf7ecfc830abb31db72a3a389c49c33d92b99e4592fbb9bfafe764c196deff" gracePeriod=15
Aug 04 18:01:57.002467 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.002389    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:57.166672 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.166643    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db17423e-c693-4630-a50d-d8b5567f2051-console-serving-cert\") pod \"db17423e-c693-4630-a50d-d8b5567f2051\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") "
Aug 04 18:01:57.166850 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.166689    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-console-config\") pod \"db17423e-c693-4630-a50d-d8b5567f2051\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") "
Aug 04 18:01:57.166850 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.166706    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-oauth-serving-cert\") pod \"db17423e-c693-4630-a50d-d8b5567f2051\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") "
Aug 04 18:01:57.166963 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.166943    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdzrw\" (UniqueName: \"kubernetes.io/projected/db17423e-c693-4630-a50d-d8b5567f2051-kube-api-access-zdzrw\") pod \"db17423e-c693-4630-a50d-d8b5567f2051\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") "
Aug 04 18:01:57.167047 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.167033    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db17423e-c693-4630-a50d-d8b5567f2051-console-oauth-config\") pod \"db17423e-c693-4630-a50d-d8b5567f2051\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") "
Aug 04 18:01:57.167123 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.167069    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-service-ca\") pod \"db17423e-c693-4630-a50d-d8b5567f2051\" (UID: \"db17423e-c693-4630-a50d-d8b5567f2051\") "
Aug 04 18:01:57.167123 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.167094    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "db17423e-c693-4630-a50d-d8b5567f2051" (UID: "db17423e-c693-4630-a50d-d8b5567f2051"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:57.167123 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.167098    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-console-config" (OuterVolumeSpecName: "console-config") pod "db17423e-c693-4630-a50d-d8b5567f2051" (UID: "db17423e-c693-4630-a50d-d8b5567f2051"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:57.167414 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.167397    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-console-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:01:57.167506 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.167417    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-oauth-serving-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:01:57.167567 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.167542    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-service-ca" (OuterVolumeSpecName: "service-ca") pod "db17423e-c693-4630-a50d-d8b5567f2051" (UID: "db17423e-c693-4630-a50d-d8b5567f2051"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:57.168947 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.168922    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db17423e-c693-4630-a50d-d8b5567f2051-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "db17423e-c693-4630-a50d-d8b5567f2051" (UID: "db17423e-c693-4630-a50d-d8b5567f2051"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:01:57.169049 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.169027    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db17423e-c693-4630-a50d-d8b5567f2051-kube-api-access-zdzrw" (OuterVolumeSpecName: "kube-api-access-zdzrw") pod "db17423e-c693-4630-a50d-d8b5567f2051" (UID: "db17423e-c693-4630-a50d-d8b5567f2051"). InnerVolumeSpecName "kube-api-access-zdzrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:01:57.169150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.169131    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db17423e-c693-4630-a50d-d8b5567f2051-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "db17423e-c693-4630-a50d-d8b5567f2051" (UID: "db17423e-c693-4630-a50d-d8b5567f2051"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:01:57.268285 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.268202    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdzrw\" (UniqueName: \"kubernetes.io/projected/db17423e-c693-4630-a50d-d8b5567f2051-kube-api-access-zdzrw\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:01:57.268285 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.268231    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db17423e-c693-4630-a50d-d8b5567f2051-console-oauth-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:01:57.268285 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.268241    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db17423e-c693-4630-a50d-d8b5567f2051-service-ca\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:01:57.268285 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.268249    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db17423e-c693-4630-a50d-d8b5567f2051-console-serving-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:01:57.739824 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.739793    2566 generic.go:356] "Generic (PLEG): container finished" podID="db17423e-c693-4630-a50d-d8b5567f2051" containerID="1acf7ecfc830abb31db72a3a389c49c33d92b99e4592fbb9bfafe764c196deff" exitCode=0
Aug 04 18:01:57.740305 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.739837    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68bfcb7466-h97jl"
Aug 04 18:01:57.740305 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.739881    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68bfcb7466-h97jl" event={"ID":"db17423e-c693-4630-a50d-d8b5567f2051","Type":"ContainerDied","Data":"1acf7ecfc830abb31db72a3a389c49c33d92b99e4592fbb9bfafe764c196deff"}
Aug 04 18:01:57.740305 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.739938    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68bfcb7466-h97jl" event={"ID":"db17423e-c693-4630-a50d-d8b5567f2051","Type":"ContainerDied","Data":"c2bbce2562801143fcf4f79365a888199ea07f546f2316110a39d8940989d94e"}
Aug 04 18:01:57.740305 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.739966    2566 scope.go:117] "RemoveContainer" containerID="1acf7ecfc830abb31db72a3a389c49c33d92b99e4592fbb9bfafe764c196deff"
Aug 04 18:01:57.748314 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.748296    2566 scope.go:117] "RemoveContainer" containerID="1acf7ecfc830abb31db72a3a389c49c33d92b99e4592fbb9bfafe764c196deff"
Aug 04 18:01:57.748603 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:01:57.748583    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1acf7ecfc830abb31db72a3a389c49c33d92b99e4592fbb9bfafe764c196deff\": container with ID starting with 1acf7ecfc830abb31db72a3a389c49c33d92b99e4592fbb9bfafe764c196deff not found: ID does not exist" containerID="1acf7ecfc830abb31db72a3a389c49c33d92b99e4592fbb9bfafe764c196deff"
Aug 04 18:01:57.748671 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.748616    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1acf7ecfc830abb31db72a3a389c49c33d92b99e4592fbb9bfafe764c196deff"} err="failed to get container status \"1acf7ecfc830abb31db72a3a389c49c33d92b99e4592fbb9bfafe764c196deff\": rpc error: code = NotFound desc = could not find container \"1acf7ecfc830abb31db72a3a389c49c33d92b99e4592fbb9bfafe764c196deff\": container with ID starting with 1acf7ecfc830abb31db72a3a389c49c33d92b99e4592fbb9bfafe764c196deff not found: ID does not exist"
Aug 04 18:01:57.761658 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.761615    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68bfcb7466-h97jl"]
Aug 04 18:01:57.766716 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:57.766673    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68bfcb7466-h97jl"]
Aug 04 18:01:59.194323 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:01:59.194288    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db17423e-c693-4630-a50d-d8b5567f2051" path="/var/lib/kubelet/pods/db17423e-c693-4630-a50d-d8b5567f2051/volumes"
Aug 04 18:02:16.800820 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:16.800783    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:16.820329 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:16.820302    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:16.835530 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:16.835505    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:30.657435 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:30.657398    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:02:30.657855 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:30.657682    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="alertmanager" containerID="cri-o://c1814e365acd691cf56fcb58666f519d8824188685c0c1c88ca18145d7a05882" gracePeriod=120
Aug 04 18:02:30.657855 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:30.657747    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="kube-rbac-proxy-metric" containerID="cri-o://d1db49503a42e1d3111a5e27c06ef6ef438e2ef3433347aeff062eca9b5d59ce" gracePeriod=120
Aug 04 18:02:30.657855 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:30.657793    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="kube-rbac-proxy-web" containerID="cri-o://d319d9d147e73b6d7ca3a6d8f0ede45c897f15b10101e923a2274d113300b03c" gracePeriod=120
Aug 04 18:02:30.658030 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:30.657834    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="prom-label-proxy" containerID="cri-o://2d3c698f80f9ea3d860b646f371c3c5e65f41ed1c8ce923e2f1989a55c4a7945" gracePeriod=120
Aug 04 18:02:30.658030 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:30.657793    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="kube-rbac-proxy" containerID="cri-o://13bb477364f0f8002ecfc4a3a03a51eb0ded223878150d7779cf7dbe8d7e163d" gracePeriod=120
Aug 04 18:02:30.658030 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:30.657936    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="config-reloader" containerID="cri-o://cc17ea317cca9b5737d581ddb96e08e5e22ca19798d21753b8cf2d49156019d8" gracePeriod=120
Aug 04 18:02:30.871756 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:30.871722    2566 generic.go:356] "Generic (PLEG): container finished" podID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerID="2d3c698f80f9ea3d860b646f371c3c5e65f41ed1c8ce923e2f1989a55c4a7945" exitCode=0
Aug 04 18:02:30.871756 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:30.871746    2566 generic.go:356] "Generic (PLEG): container finished" podID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerID="13bb477364f0f8002ecfc4a3a03a51eb0ded223878150d7779cf7dbe8d7e163d" exitCode=0
Aug 04 18:02:30.871756 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:30.871753    2566 generic.go:356] "Generic (PLEG): container finished" podID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerID="cc17ea317cca9b5737d581ddb96e08e5e22ca19798d21753b8cf2d49156019d8" exitCode=0
Aug 04 18:02:30.871756 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:30.871758    2566 generic.go:356] "Generic (PLEG): container finished" podID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerID="c1814e365acd691cf56fcb58666f519d8824188685c0c1c88ca18145d7a05882" exitCode=0
Aug 04 18:02:30.872012 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:30.871793    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a","Type":"ContainerDied","Data":"2d3c698f80f9ea3d860b646f371c3c5e65f41ed1c8ce923e2f1989a55c4a7945"}
Aug 04 18:02:30.872012 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:30.871864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a","Type":"ContainerDied","Data":"13bb477364f0f8002ecfc4a3a03a51eb0ded223878150d7779cf7dbe8d7e163d"}
Aug 04 18:02:30.872012 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:30.871892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a","Type":"ContainerDied","Data":"cc17ea317cca9b5737d581ddb96e08e5e22ca19798d21753b8cf2d49156019d8"}
Aug 04 18:02:30.872012 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:30.871916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a","Type":"ContainerDied","Data":"c1814e365acd691cf56fcb58666f519d8824188685c0c1c88ca18145d7a05882"}
Aug 04 18:02:31.879766 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.879726    2566 generic.go:356] "Generic (PLEG): container finished" podID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerID="d1db49503a42e1d3111a5e27c06ef6ef438e2ef3433347aeff062eca9b5d59ce" exitCode=0
Aug 04 18:02:31.879766 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.879756    2566 generic.go:356] "Generic (PLEG): container finished" podID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerID="d319d9d147e73b6d7ca3a6d8f0ede45c897f15b10101e923a2274d113300b03c" exitCode=0
Aug 04 18:02:31.880145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.879805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a","Type":"ContainerDied","Data":"d1db49503a42e1d3111a5e27c06ef6ef438e2ef3433347aeff062eca9b5d59ce"}
Aug 04 18:02:31.880145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.879929    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a","Type":"ContainerDied","Data":"d319d9d147e73b6d7ca3a6d8f0ede45c897f15b10101e923a2274d113300b03c"}
Aug 04 18:02:31.912419 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.912389    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:31.973916 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.973834    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-cluster-tls-config\") pod \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") "
Aug 04 18:02:31.973916 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.973881    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-alertmanager-main-db\") pod \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") "
Aug 04 18:02:31.974128 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.973920    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-tls-assets\") pod \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") "
Aug 04 18:02:31.974128 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.973950    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-web-config\") pod \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") "
Aug 04 18:02:31.974128 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.973986    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy\") pod \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") "
Aug 04 18:02:31.974128 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.974055    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-config-volume\") pod \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") "
Aug 04 18:02:31.974128 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.974091    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy-web\") pod \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") "
Aug 04 18:02:31.974128 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.974119    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-metrics-client-ca\") pod \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") "
Aug 04 18:02:31.974453 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.974146    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp6mg\" (UniqueName: \"kubernetes.io/projected/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-kube-api-access-hp6mg\") pod \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") "
Aug 04 18:02:31.974453 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.974170    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") "
Aug 04 18:02:31.974453 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.974201    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-config-out\") pod \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") "
Aug 04 18:02:31.974453 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.974243    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-alertmanager-trusted-ca-bundle\") pod \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") "
Aug 04 18:02:31.974453 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.974260    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" (UID: "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:02:31.974453 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.974281    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-main-tls\") pod \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\" (UID: \"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a\") "
Aug 04 18:02:31.974756 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.974624    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-alertmanager-main-db\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:31.976206 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.976169    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" (UID: "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:31.976367 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.976223    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" (UID: "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:31.977931 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.977905    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" (UID: "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:31.978040 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.977966    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" (UID: "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:31.978040 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.978021    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-config-volume" (OuterVolumeSpecName: "config-volume") pod "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" (UID: "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:31.978202 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.978169    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" (UID: "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:31.978394 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.978337    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" (UID: "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:31.978619 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.978586    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" (UID: "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:31.978932 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.978910    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-kube-api-access-hp6mg" (OuterVolumeSpecName: "kube-api-access-hp6mg") pod "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" (UID: "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a"). InnerVolumeSpecName "kube-api-access-hp6mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:31.980196 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.980176    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-config-out" (OuterVolumeSpecName: "config-out") pod "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" (UID: "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:02:31.984114 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.984086    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" (UID: "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:31.991108 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:31.991078    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-web-config" (OuterVolumeSpecName: "web-config") pod "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" (UID: "688b3da2-e0c3-4b9d-ac5d-d8410e5d065a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:32.075580 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.075534    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:32.075580 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.075576    2566 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-config-volume\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:32.075580 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.075588    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:32.075580 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.075597    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-metrics-client-ca\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:32.075860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.075606    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp6mg\" (UniqueName: \"kubernetes.io/projected/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-kube-api-access-hp6mg\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:32.075860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.075615    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:32.075860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.075624    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-config-out\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:32.075860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.075633    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:32.075860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.075641    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-secret-alertmanager-main-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:32.075860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.075650    2566 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-cluster-tls-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:32.075860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.075659    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-tls-assets\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:32.075860 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.075667    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a-web-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:32.886861 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.886822    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"688b3da2-e0c3-4b9d-ac5d-d8410e5d065a","Type":"ContainerDied","Data":"40360cf9c8d4fb839eaa309e0285544875fe4b7d2aeb6c95e95629eb8d469e16"}
Aug 04 18:02:32.887317 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.886875    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:32.887317 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.886912    2566 scope.go:117] "RemoveContainer" containerID="2d3c698f80f9ea3d860b646f371c3c5e65f41ed1c8ce923e2f1989a55c4a7945"
Aug 04 18:02:32.894657 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.894571    2566 scope.go:117] "RemoveContainer" containerID="d1db49503a42e1d3111a5e27c06ef6ef438e2ef3433347aeff062eca9b5d59ce"
Aug 04 18:02:32.902249 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.902232    2566 scope.go:117] "RemoveContainer" containerID="13bb477364f0f8002ecfc4a3a03a51eb0ded223878150d7779cf7dbe8d7e163d"
Aug 04 18:02:32.909579 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.909558    2566 scope.go:117] "RemoveContainer" containerID="d319d9d147e73b6d7ca3a6d8f0ede45c897f15b10101e923a2274d113300b03c"
Aug 04 18:02:32.915042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.915020    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:02:32.918212 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.918190    2566 scope.go:117] "RemoveContainer" containerID="cc17ea317cca9b5737d581ddb96e08e5e22ca19798d21753b8cf2d49156019d8"
Aug 04 18:02:32.920111 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.920089    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:02:32.925715 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.925698    2566 scope.go:117] "RemoveContainer" containerID="c1814e365acd691cf56fcb58666f519d8824188685c0c1c88ca18145d7a05882"
Aug 04 18:02:32.932768 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.932754    2566 scope.go:117] "RemoveContainer" containerID="3c3b0047649f5d3d53c4b08d73a0855948512f65e65d6185564553c2d2728aac"
Aug 04 18:02:32.950522 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950494    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:02:32.950811 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950800    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="kube-rbac-proxy"
Aug 04 18:02:32.950866 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950814    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="kube-rbac-proxy"
Aug 04 18:02:32.950866 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950824    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="kube-rbac-proxy-web"
Aug 04 18:02:32.950866 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950831    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="kube-rbac-proxy-web"
Aug 04 18:02:32.950866 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950838    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="prom-label-proxy"
Aug 04 18:02:32.950866 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950844    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="prom-label-proxy"
Aug 04 18:02:32.950866 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950851    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="alertmanager"
Aug 04 18:02:32.950866 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950857    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="alertmanager"
Aug 04 18:02:32.950866 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950865    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="config-reloader"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950872    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="config-reloader"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950877    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="416d258c-7d82-4310-84f3-dc32908b0a4e" containerName="console"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950883    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="416d258c-7d82-4310-84f3-dc32908b0a4e" containerName="console"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950888    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="init-config-reloader"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950893    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="init-config-reloader"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950900    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="kube-rbac-proxy-metric"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950905    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="kube-rbac-proxy-metric"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950919    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db17423e-c693-4630-a50d-d8b5567f2051" containerName="console"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950924    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="db17423e-c693-4630-a50d-d8b5567f2051" containerName="console"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950971    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="db17423e-c693-4630-a50d-d8b5567f2051" containerName="console"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950977    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="alertmanager"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950985    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="416d258c-7d82-4310-84f3-dc32908b0a4e" containerName="console"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950992    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="kube-rbac-proxy-metric"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.950997    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="kube-rbac-proxy"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.951003    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="prom-label-proxy"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.951009    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="kube-rbac-proxy-web"
Aug 04 18:02:32.951092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.951014    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" containerName="config-reloader"
Aug 04 18:02:32.955451 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.955426    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:32.957988 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.957965    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Aug 04 18:02:32.958110 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.957965    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Aug 04 18:02:32.958110 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.957968    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4q9wp\""
Aug 04 18:02:32.958110 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.957968    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Aug 04 18:02:32.958328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.958299    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Aug 04 18:02:32.958473 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.958404    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Aug 04 18:02:32.958631 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.958611    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Aug 04 18:02:32.958631 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.958619    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Aug 04 18:02:32.958825 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.958652    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Aug 04 18:02:32.963610 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.963590    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Aug 04 18:02:32.975715 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:32.975644    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:02:33.085225 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.085188    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-web-config\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.085417 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.085233    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.085417 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.085271    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.085417 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.085298    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.085417 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.085321    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82af722f-f621-4ca4-9586-c6fbf84591b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.085628 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.085428    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/82af722f-f621-4ca4-9586-c6fbf84591b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.085628 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.085490    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcghw\" (UniqueName: \"kubernetes.io/projected/82af722f-f621-4ca4-9586-c6fbf84591b2-kube-api-access-zcghw\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.085628 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.085520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.085628 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.085558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82af722f-f621-4ca4-9586-c6fbf84591b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.085628 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.085611    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82af722f-f621-4ca4-9586-c6fbf84591b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.085829 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.085657    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82af722f-f621-4ca4-9586-c6fbf84591b2-config-out\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.085829 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.085692    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.085829 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.085748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.186612 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.186575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82af722f-f621-4ca4-9586-c6fbf84591b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.186794 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.186620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82af722f-f621-4ca4-9586-c6fbf84591b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.186794 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.186641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82af722f-f621-4ca4-9586-c6fbf84591b2-config-out\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.186794 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.186665    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.186794 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.186699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.186794 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.186728    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-web-config\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.186794 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.186755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.187085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.186884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.187085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.186909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.187085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.186932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82af722f-f621-4ca4-9586-c6fbf84591b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.187085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.186963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/82af722f-f621-4ca4-9586-c6fbf84591b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.187085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.187004    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcghw\" (UniqueName: \"kubernetes.io/projected/82af722f-f621-4ca4-9586-c6fbf84591b2-kube-api-access-zcghw\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.187085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.187027    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.187730 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.187461    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82af722f-f621-4ca4-9586-c6fbf84591b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.189079 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.188717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82af722f-f621-4ca4-9586-c6fbf84591b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.189079 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.188981    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/82af722f-f621-4ca4-9586-c6fbf84591b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.189622 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.189597    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82af722f-f621-4ca4-9586-c6fbf84591b2-config-out\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.190600 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.190099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.190600 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.190189    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82af722f-f621-4ca4-9586-c6fbf84591b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.190600 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.190222    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.190600 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.190460    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.190600 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.190564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.191252 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.191232    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.191622 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.191603    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.191771 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.191752    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82af722f-f621-4ca4-9586-c6fbf84591b2-web-config\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.195170 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.195147    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="688b3da2-e0c3-4b9d-ac5d-d8410e5d065a" path="/var/lib/kubelet/pods/688b3da2-e0c3-4b9d-ac5d-d8410e5d065a/volumes"
Aug 04 18:02:33.198453 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.198434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcghw\" (UniqueName: \"kubernetes.io/projected/82af722f-f621-4ca4-9586-c6fbf84591b2-kube-api-access-zcghw\") pod \"alertmanager-main-0\" (UID: \"82af722f-f621-4ca4-9586-c6fbf84591b2\") " pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.266748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.266665    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Aug 04 18:02:33.418344 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.415279    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Aug 04 18:02:33.893277 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.893242    2566 generic.go:356] "Generic (PLEG): container finished" podID="82af722f-f621-4ca4-9586-c6fbf84591b2" containerID="ae347592fe22dadad59588b2ec68796b632e07237a3e9c271f41e18243460b95" exitCode=0
Aug 04 18:02:33.893673 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.893338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82af722f-f621-4ca4-9586-c6fbf84591b2","Type":"ContainerDied","Data":"ae347592fe22dadad59588b2ec68796b632e07237a3e9c271f41e18243460b95"}
Aug 04 18:02:33.893673 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:33.893419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82af722f-f621-4ca4-9586-c6fbf84591b2","Type":"ContainerStarted","Data":"a44d9968c86b6789a604ffec8b79fb199e65fa4def316647f355940c872264c8"}
Aug 04 18:02:34.901748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:34.901705    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82af722f-f621-4ca4-9586-c6fbf84591b2","Type":"ContainerStarted","Data":"ba0a12fd50d9e15a882d997824074c5d95dc1a92d2dff23ef6a4e6cdf4d66dcc"}
Aug 04 18:02:34.902183 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:34.901793    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82af722f-f621-4ca4-9586-c6fbf84591b2","Type":"ContainerStarted","Data":"1adb1671a63b574fef036809989e1e2dc14ba75b608e81f7fcc8384f112783a4"}
Aug 04 18:02:34.902183 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:34.901827    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82af722f-f621-4ca4-9586-c6fbf84591b2","Type":"ContainerStarted","Data":"2ee9dd51053671a3bc35403c3eb332eb629b7631108523f319ef94b4f210a8cd"}
Aug 04 18:02:34.902183 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:34.901860    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82af722f-f621-4ca4-9586-c6fbf84591b2","Type":"ContainerStarted","Data":"26c6153f070c1751681c158c266da2ed5e58666b397abf2f4c8c894019dd8d15"}
Aug 04 18:02:34.902183 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:34.901882    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82af722f-f621-4ca4-9586-c6fbf84591b2","Type":"ContainerStarted","Data":"5bb047b9889bf97e174017a904e22d5ec3e820cbeee15d5c02dc441b78a4b043"}
Aug 04 18:02:34.902183 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:34.901905    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82af722f-f621-4ca4-9586-c6fbf84591b2","Type":"ContainerStarted","Data":"f1772679e59a5fbfb0c7f8c9122331aa88d2cdb69db3ee3c041d9b5971bce19e"}
Aug 04 18:02:34.938552 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:34.938484    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.92388474 podStartE2EDuration="2.938463433s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.578693ms" isStatefulPod=false podCreationTimestamp="2026-08-04 18:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:34.935889389 +0000 UTC m=+154.226503579" watchObservedRunningTime="2026-08-04 18:02:34.938463433 +0000 UTC m=+154.229077605"
Aug 04 18:02:34.961671 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:34.961642    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:02:34.962029 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:34.961994    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="prometheus" containerID="cri-o://988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0" gracePeriod=600
Aug 04 18:02:34.962029 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:34.962011    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="kube-rbac-proxy-web" containerID="cri-o://4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c" gracePeriod=600
Aug 04 18:02:34.962255 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:34.962051    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="config-reloader" containerID="cri-o://1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83" gracePeriod=600
Aug 04 18:02:34.962255 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:34.962077    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131" gracePeriod=600
Aug 04 18:02:34.962255 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:34.962020    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="kube-rbac-proxy" containerID="cri-o://9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460" gracePeriod=600
Aug 04 18:02:34.963021 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:34.962997    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="thanos-sidecar" containerID="cri-o://05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e" gracePeriod=600
Aug 04 18:02:35.222293 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.222247    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:35.305323 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.305295    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-1\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.305515 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.305335    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-grpc-tls\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.305515 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.305371    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-trusted-ca-bundle\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.305515 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.305393    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-config\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.305767 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.305571    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f63bfef0-7395-4857-9f3b-a59161a9703b-config-out\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.305767 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.305626    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-0\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.305767 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.305670    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.305767 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.305682    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:35.305767 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.305714    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f63bfef0-7395-4857-9f3b-a59161a9703b-tls-assets\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.305767 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.305741    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:35.306056 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.305824    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-metrics-client-certs\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.306128 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.306104    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-metrics-client-ca\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.306184 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.306167    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-db\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.306262 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.306210    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-serving-certs-ca-bundle\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.306262 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.306240    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-kube-rbac-proxy\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.306383 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.306271    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-2\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.306383 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.306307    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-thanos-prometheus-http-client-file\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.306383 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.306333    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfsc4\" (UniqueName: \"kubernetes.io/projected/f63bfef0-7395-4857-9f3b-a59161a9703b-kube-api-access-wfsc4\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.306383 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.306378    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-kubelet-serving-ca-bundle\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.306593 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.306420    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-web-config\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.306593 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.306452    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.306593 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.306484    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-tls\") pod \"f63bfef0-7395-4857-9f3b-a59161a9703b\" (UID: \"f63bfef0-7395-4857-9f3b-a59161a9703b\") "
Aug 04 18:02:35.306783 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.306755    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:35.306904 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.306887    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.306958 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.306910    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.306958 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.306926    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.307267 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.307238    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:35.308175 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.308144    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:35.308175 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.308158    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-config" (OuterVolumeSpecName: "config") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:35.308775 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.308513    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:35.308775 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.308545    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:35.308945 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.308915    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:35.309013 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.308975    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:35.309074 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.309027    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63bfef0-7395-4857-9f3b-a59161a9703b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:35.309853 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.309825    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:02:35.310124 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.310095    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:35.310333 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.310307    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63bfef0-7395-4857-9f3b-a59161a9703b-kube-api-access-wfsc4" (OuterVolumeSpecName: "kube-api-access-wfsc4") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "kube-api-access-wfsc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:35.311065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.311041    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63bfef0-7395-4857-9f3b-a59161a9703b-config-out" (OuterVolumeSpecName: "config-out") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:02:35.311246 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.311223    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:35.311728 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.311696    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:35.312071 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.312046    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:35.312447 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.312427    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:35.322842 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.322820    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-web-config" (OuterVolumeSpecName: "web-config") pod "f63bfef0-7395-4857-9f3b-a59161a9703b" (UID: "f63bfef0-7395-4857-9f3b-a59161a9703b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:35.408100 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408068    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-metrics-client-ca\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.408100 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408097    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-db\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.408327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408111    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.408327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408124    2566 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-kube-rbac-proxy\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.408327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408136    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.408327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408147    2566 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.408327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408159    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfsc4\" (UniqueName: \"kubernetes.io/projected/f63bfef0-7395-4857-9f3b-a59161a9703b-kube-api-access-wfsc4\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.408327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408172    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f63bfef0-7395-4857-9f3b-a59161a9703b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.408327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408185    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-web-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.408327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408196    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.408327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408208    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.408327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408219    2566 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-grpc-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.408327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408231    2566 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.408327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408243    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f63bfef0-7395-4857-9f3b-a59161a9703b-config-out\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.408327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408254    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.408327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408266    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f63bfef0-7395-4857-9f3b-a59161a9703b-tls-assets\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.408327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.408277    2566 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f63bfef0-7395-4857-9f3b-a59161a9703b-secret-metrics-client-certs\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:02:35.909192 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.909160    2566 generic.go:356] "Generic (PLEG): container finished" podID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerID="73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131" exitCode=0
Aug 04 18:02:35.909192 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.909187    2566 generic.go:356] "Generic (PLEG): container finished" podID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerID="9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460" exitCode=0
Aug 04 18:02:35.909192 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.909197    2566 generic.go:356] "Generic (PLEG): container finished" podID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerID="4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c" exitCode=0
Aug 04 18:02:35.909682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.909205    2566 generic.go:356] "Generic (PLEG): container finished" podID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerID="05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e" exitCode=0
Aug 04 18:02:35.909682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.909214    2566 generic.go:356] "Generic (PLEG): container finished" podID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerID="1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83" exitCode=0
Aug 04 18:02:35.909682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.909222    2566 generic.go:356] "Generic (PLEG): container finished" podID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerID="988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0" exitCode=0
Aug 04 18:02:35.909682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.909250    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:35.909682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.909250    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f63bfef0-7395-4857-9f3b-a59161a9703b","Type":"ContainerDied","Data":"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131"}
Aug 04 18:02:35.909682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.909341    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f63bfef0-7395-4857-9f3b-a59161a9703b","Type":"ContainerDied","Data":"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460"}
Aug 04 18:02:35.909682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.909389    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f63bfef0-7395-4857-9f3b-a59161a9703b","Type":"ContainerDied","Data":"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c"}
Aug 04 18:02:35.909682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.909417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f63bfef0-7395-4857-9f3b-a59161a9703b","Type":"ContainerDied","Data":"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e"}
Aug 04 18:02:35.909682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.909441    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f63bfef0-7395-4857-9f3b-a59161a9703b","Type":"ContainerDied","Data":"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83"}
Aug 04 18:02:35.909682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.909433    2566 scope.go:117] "RemoveContainer" containerID="73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131"
Aug 04 18:02:35.909682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.909465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f63bfef0-7395-4857-9f3b-a59161a9703b","Type":"ContainerDied","Data":"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0"}
Aug 04 18:02:35.909682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.909490    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f63bfef0-7395-4857-9f3b-a59161a9703b","Type":"ContainerDied","Data":"7a748fe1f9abbc96aaa93c67052df89e1da41f97485de638f3369c4722549c6f"}
Aug 04 18:02:35.917741 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.917720    2566 scope.go:117] "RemoveContainer" containerID="9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460"
Aug 04 18:02:35.924827 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.924813    2566 scope.go:117] "RemoveContainer" containerID="4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c"
Aug 04 18:02:35.931830 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.931813    2566 scope.go:117] "RemoveContainer" containerID="05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e"
Aug 04 18:02:35.936256 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.936234    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:02:35.939700 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.939565    2566 scope.go:117] "RemoveContainer" containerID="1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83"
Aug 04 18:02:35.941499 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.941479    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:02:35.946638 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.946622    2566 scope.go:117] "RemoveContainer" containerID="988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0"
Aug 04 18:02:35.953705 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.953691    2566 scope.go:117] "RemoveContainer" containerID="a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885"
Aug 04 18:02:35.960143 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.960114    2566 scope.go:117] "RemoveContainer" containerID="73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131"
Aug 04 18:02:35.960431 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:02:35.960410    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131\": container with ID starting with 73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131 not found: ID does not exist" containerID="73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131"
Aug 04 18:02:35.960515 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.960444    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131"} err="failed to get container status \"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131\": rpc error: code = NotFound desc = could not find container \"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131\": container with ID starting with 73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131 not found: ID does not exist"
Aug 04 18:02:35.960515 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.960469    2566 scope.go:117] "RemoveContainer" containerID="9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460"
Aug 04 18:02:35.960723 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:02:35.960708    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460\": container with ID starting with 9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460 not found: ID does not exist" containerID="9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460"
Aug 04 18:02:35.960789 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.960731    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460"} err="failed to get container status \"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460\": rpc error: code = NotFound desc = could not find container \"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460\": container with ID starting with 9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460 not found: ID does not exist"
Aug 04 18:02:35.960789 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.960750    2566 scope.go:117] "RemoveContainer" containerID="4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c"
Aug 04 18:02:35.960979 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:02:35.960963    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c\": container with ID starting with 4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c not found: ID does not exist" containerID="4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c"
Aug 04 18:02:35.961019 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.960985    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c"} err="failed to get container status \"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c\": rpc error: code = NotFound desc = could not find container \"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c\": container with ID starting with 4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c not found: ID does not exist"
Aug 04 18:02:35.961019 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.961000    2566 scope.go:117] "RemoveContainer" containerID="05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e"
Aug 04 18:02:35.961226 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:02:35.961209    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e\": container with ID starting with 05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e not found: ID does not exist" containerID="05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e"
Aug 04 18:02:35.961280 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.961231    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e"} err="failed to get container status \"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e\": rpc error: code = NotFound desc = could not find container \"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e\": container with ID starting with 05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e not found: ID does not exist"
Aug 04 18:02:35.961280 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.961247    2566 scope.go:117] "RemoveContainer" containerID="1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83"
Aug 04 18:02:35.961443 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:02:35.961426    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83\": container with ID starting with 1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83 not found: ID does not exist" containerID="1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83"
Aug 04 18:02:35.961501 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.961450    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83"} err="failed to get container status \"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83\": rpc error: code = NotFound desc = could not find container \"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83\": container with ID starting with 1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83 not found: ID does not exist"
Aug 04 18:02:35.961501 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.961470    2566 scope.go:117] "RemoveContainer" containerID="988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0"
Aug 04 18:02:35.961675 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:02:35.961660    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0\": container with ID starting with 988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0 not found: ID does not exist" containerID="988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0"
Aug 04 18:02:35.961714 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.961680    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0"} err="failed to get container status \"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0\": rpc error: code = NotFound desc = could not find container \"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0\": container with ID starting with 988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0 not found: ID does not exist"
Aug 04 18:02:35.961714 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.961694    2566 scope.go:117] "RemoveContainer" containerID="a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885"
Aug 04 18:02:35.961907 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:02:35.961887    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885\": container with ID starting with a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885 not found: ID does not exist" containerID="a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885"
Aug 04 18:02:35.961963 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.961913    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885"} err="failed to get container status \"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885\": rpc error: code = NotFound desc = could not find container \"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885\": container with ID starting with a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885 not found: ID does not exist"
Aug 04 18:02:35.961963 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.961931    2566 scope.go:117] "RemoveContainer" containerID="73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131"
Aug 04 18:02:35.962166 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.962150    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131"} err="failed to get container status \"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131\": rpc error: code = NotFound desc = could not find container \"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131\": container with ID starting with 73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131 not found: ID does not exist"
Aug 04 18:02:35.962212 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.962167    2566 scope.go:117] "RemoveContainer" containerID="9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460"
Aug 04 18:02:35.962469 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.962374    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460"} err="failed to get container status \"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460\": rpc error: code = NotFound desc = could not find container \"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460\": container with ID starting with 9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460 not found: ID does not exist"
Aug 04 18:02:35.962469 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.962393    2566 scope.go:117] "RemoveContainer" containerID="4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c"
Aug 04 18:02:35.962637 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.962623    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c"} err="failed to get container status \"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c\": rpc error: code = NotFound desc = could not find container \"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c\": container with ID starting with 4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c not found: ID does not exist"
Aug 04 18:02:35.962637 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.962637    2566 scope.go:117] "RemoveContainer" containerID="05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e"
Aug 04 18:02:35.962852 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.962831    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e"} err="failed to get container status \"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e\": rpc error: code = NotFound desc = could not find container \"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e\": container with ID starting with 05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e not found: ID does not exist"
Aug 04 18:02:35.962892 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.962852    2566 scope.go:117] "RemoveContainer" containerID="1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83"
Aug 04 18:02:35.963052 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.963035    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83"} err="failed to get container status \"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83\": rpc error: code = NotFound desc = could not find container \"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83\": container with ID starting with 1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83 not found: ID does not exist"
Aug 04 18:02:35.963088 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.963052    2566 scope.go:117] "RemoveContainer" containerID="988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0"
Aug 04 18:02:35.963240 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.963225    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0"} err="failed to get container status \"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0\": rpc error: code = NotFound desc = could not find container \"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0\": container with ID starting with 988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0 not found: ID does not exist"
Aug 04 18:02:35.963284 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.963240    2566 scope.go:117] "RemoveContainer" containerID="a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885"
Aug 04 18:02:35.963479 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.963458    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885"} err="failed to get container status \"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885\": rpc error: code = NotFound desc = could not find container \"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885\": container with ID starting with a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885 not found: ID does not exist"
Aug 04 18:02:35.963520 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.963481    2566 scope.go:117] "RemoveContainer" containerID="73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131"
Aug 04 18:02:35.963703 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.963686    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131"} err="failed to get container status \"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131\": rpc error: code = NotFound desc = could not find container \"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131\": container with ID starting with 73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131 not found: ID does not exist"
Aug 04 18:02:35.963744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.963705    2566 scope.go:117] "RemoveContainer" containerID="9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460"
Aug 04 18:02:35.963922 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.963900    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460"} err="failed to get container status \"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460\": rpc error: code = NotFound desc = could not find container \"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460\": container with ID starting with 9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460 not found: ID does not exist"
Aug 04 18:02:35.963991 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.963922    2566 scope.go:117] "RemoveContainer" containerID="4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c"
Aug 04 18:02:35.964173 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.964154    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c"} err="failed to get container status \"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c\": rpc error: code = NotFound desc = could not find container \"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c\": container with ID starting with 4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c not found: ID does not exist"
Aug 04 18:02:35.964232 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.964173    2566 scope.go:117] "RemoveContainer" containerID="05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e"
Aug 04 18:02:35.964451 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.964432    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e"} err="failed to get container status \"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e\": rpc error: code = NotFound desc = could not find container \"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e\": container with ID starting with 05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e not found: ID does not exist"
Aug 04 18:02:35.964451 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.964451    2566 scope.go:117] "RemoveContainer" containerID="1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83"
Aug 04 18:02:35.964642 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.964626    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83"} err="failed to get container status \"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83\": rpc error: code = NotFound desc = could not find container \"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83\": container with ID starting with 1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83 not found: ID does not exist"
Aug 04 18:02:35.964683 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.964643    2566 scope.go:117] "RemoveContainer" containerID="988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0"
Aug 04 18:02:35.964831 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.964815    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0"} err="failed to get container status \"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0\": rpc error: code = NotFound desc = could not find container \"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0\": container with ID starting with 988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0 not found: ID does not exist"
Aug 04 18:02:35.964869 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.964831    2566 scope.go:117] "RemoveContainer" containerID="a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885"
Aug 04 18:02:35.965018 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.965002    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885"} err="failed to get container status \"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885\": rpc error: code = NotFound desc = could not find container \"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885\": container with ID starting with a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885 not found: ID does not exist"
Aug 04 18:02:35.965060 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.965019    2566 scope.go:117] "RemoveContainer" containerID="73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131"
Aug 04 18:02:35.965190 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.965174    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131"} err="failed to get container status \"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131\": rpc error: code = NotFound desc = could not find container \"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131\": container with ID starting with 73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131 not found: ID does not exist"
Aug 04 18:02:35.965233 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.965190    2566 scope.go:117] "RemoveContainer" containerID="9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460"
Aug 04 18:02:35.965404 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.965386    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460"} err="failed to get container status \"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460\": rpc error: code = NotFound desc = could not find container \"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460\": container with ID starting with 9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460 not found: ID does not exist"
Aug 04 18:02:35.965466 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.965407    2566 scope.go:117] "RemoveContainer" containerID="4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c"
Aug 04 18:02:35.965648 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.965631    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c"} err="failed to get container status \"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c\": rpc error: code = NotFound desc = could not find container \"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c\": container with ID starting with 4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c not found: ID does not exist"
Aug 04 18:02:35.965703 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.965650    2566 scope.go:117] "RemoveContainer" containerID="05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e"
Aug 04 18:02:35.965880 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.965863    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e"} err="failed to get container status \"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e\": rpc error: code = NotFound desc = could not find container \"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e\": container with ID starting with 05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e not found: ID does not exist"
Aug 04 18:02:35.965921 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.965890    2566 scope.go:117] "RemoveContainer" containerID="1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83"
Aug 04 18:02:35.966075 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.966057    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83"} err="failed to get container status \"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83\": rpc error: code = NotFound desc = could not find container \"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83\": container with ID starting with 1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83 not found: ID does not exist"
Aug 04 18:02:35.966135 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.966076    2566 scope.go:117] "RemoveContainer" containerID="988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0"
Aug 04 18:02:35.966300 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.966281    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0"} err="failed to get container status \"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0\": rpc error: code = NotFound desc = could not find container \"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0\": container with ID starting with 988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0 not found: ID does not exist"
Aug 04 18:02:35.966300 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.966298    2566 scope.go:117] "RemoveContainer" containerID="a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885"
Aug 04 18:02:35.966574 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.966549    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885"} err="failed to get container status \"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885\": rpc error: code = NotFound desc = could not find container \"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885\": container with ID starting with a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885 not found: ID does not exist"
Aug 04 18:02:35.966660 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.966575    2566 scope.go:117] "RemoveContainer" containerID="73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131"
Aug 04 18:02:35.966988 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.966966    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131"} err="failed to get container status \"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131\": rpc error: code = NotFound desc = could not find container \"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131\": container with ID starting with 73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131 not found: ID does not exist"
Aug 04 18:02:35.966988 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.966987    2566 scope.go:117] "RemoveContainer" containerID="9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460"
Aug 04 18:02:35.967254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.967235    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460"} err="failed to get container status \"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460\": rpc error: code = NotFound desc = could not find container \"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460\": container with ID starting with 9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460 not found: ID does not exist"
Aug 04 18:02:35.967294 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.967255    2566 scope.go:117] "RemoveContainer" containerID="4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c"
Aug 04 18:02:35.967510 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.967493    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c"} err="failed to get container status \"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c\": rpc error: code = NotFound desc = could not find container \"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c\": container with ID starting with 4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c not found: ID does not exist"
Aug 04 18:02:35.967550 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.967511    2566 scope.go:117] "RemoveContainer" containerID="05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e"
Aug 04 18:02:35.967740 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.967723    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e"} err="failed to get container status \"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e\": rpc error: code = NotFound desc = could not find container \"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e\": container with ID starting with 05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e not found: ID does not exist"
Aug 04 18:02:35.967779 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.967741    2566 scope.go:117] "RemoveContainer" containerID="1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83"
Aug 04 18:02:35.967945 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.967930    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83"} err="failed to get container status \"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83\": rpc error: code = NotFound desc = could not find container \"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83\": container with ID starting with 1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83 not found: ID does not exist"
Aug 04 18:02:35.967983 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.967945    2566 scope.go:117] "RemoveContainer" containerID="988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0"
Aug 04 18:02:35.968132 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.968119    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0"} err="failed to get container status \"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0\": rpc error: code = NotFound desc = could not find container \"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0\": container with ID starting with 988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0 not found: ID does not exist"
Aug 04 18:02:35.968171 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.968133    2566 scope.go:117] "RemoveContainer" containerID="a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885"
Aug 04 18:02:35.968318 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.968297    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885"} err="failed to get container status \"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885\": rpc error: code = NotFound desc = could not find container \"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885\": container with ID starting with a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885 not found: ID does not exist"
Aug 04 18:02:35.968375 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.968319    2566 scope.go:117] "RemoveContainer" containerID="73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131"
Aug 04 18:02:35.968546 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.968531    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131"} err="failed to get container status \"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131\": rpc error: code = NotFound desc = could not find container \"73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131\": container with ID starting with 73af52e8a36e78ee9522de9cb6d0262e984e3101c5ad0a81d73f25cd89b09131 not found: ID does not exist"
Aug 04 18:02:35.968583 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.968547    2566 scope.go:117] "RemoveContainer" containerID="9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460"
Aug 04 18:02:35.968760 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.968742    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460"} err="failed to get container status \"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460\": rpc error: code = NotFound desc = could not find container \"9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460\": container with ID starting with 9febc38886a348e4957ab6b2584003b6b519a1b30c60ccb9365831a7ac065460 not found: ID does not exist"
Aug 04 18:02:35.968796 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.968761    2566 scope.go:117] "RemoveContainer" containerID="4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c"
Aug 04 18:02:35.968990 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.968969    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c"} err="failed to get container status \"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c\": rpc error: code = NotFound desc = could not find container \"4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c\": container with ID starting with 4652f54b31729f129903d9625650c702bd6007942914183f97edae41f555443c not found: ID does not exist"
Aug 04 18:02:35.969048 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.968990    2566 scope.go:117] "RemoveContainer" containerID="05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e"
Aug 04 18:02:35.969195 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.969180    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e"} err="failed to get container status \"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e\": rpc error: code = NotFound desc = could not find container \"05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e\": container with ID starting with 05909c3794a7a58eefa06e93aa2401349d294d9f52505839b370f69ee74de94e not found: ID does not exist"
Aug 04 18:02:35.969239 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.969196    2566 scope.go:117] "RemoveContainer" containerID="1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83"
Aug 04 18:02:35.969407 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.969390    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83"} err="failed to get container status \"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83\": rpc error: code = NotFound desc = could not find container \"1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83\": container with ID starting with 1eb8b8a290937ca9803a9fa47dcd6c4662bf2639f8231408bc6c695639e73c83 not found: ID does not exist"
Aug 04 18:02:35.969467 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.969408    2566 scope.go:117] "RemoveContainer" containerID="988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0"
Aug 04 18:02:35.969627 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.969609    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0"} err="failed to get container status \"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0\": rpc error: code = NotFound desc = could not find container \"988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0\": container with ID starting with 988d3e046c6b92b4ce8033361fa9916be0ac995e3134ef4511f8f1f681127cf0 not found: ID does not exist"
Aug 04 18:02:35.969676 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.969628    2566 scope.go:117] "RemoveContainer" containerID="a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885"
Aug 04 18:02:35.969835 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.969818    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885"} err="failed to get container status \"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885\": rpc error: code = NotFound desc = could not find container \"a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885\": container with ID starting with a34ac75b9f0f4efb12b48bf8bb962251ba93ee8faa953b6b79cf03b671060885 not found: ID does not exist"
Aug 04 18:02:35.977707 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.977657    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:02:35.977958 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.977948    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="config-reloader"
Aug 04 18:02:35.978001 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.977960    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="config-reloader"
Aug 04 18:02:35.978001 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.977974    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="init-config-reloader"
Aug 04 18:02:35.978001 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.977979    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="init-config-reloader"
Aug 04 18:02:35.978001 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.977988    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="thanos-sidecar"
Aug 04 18:02:35.978001 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.977993    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="thanos-sidecar"
Aug 04 18:02:35.978145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.978009    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="kube-rbac-proxy-web"
Aug 04 18:02:35.978145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.978014    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="kube-rbac-proxy-web"
Aug 04 18:02:35.978145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.978020    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="kube-rbac-proxy"
Aug 04 18:02:35.978145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.978025    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="kube-rbac-proxy"
Aug 04 18:02:35.978145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.978032    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="prometheus"
Aug 04 18:02:35.978145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.978037    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="prometheus"
Aug 04 18:02:35.978145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.978043    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="kube-rbac-proxy-thanos"
Aug 04 18:02:35.978145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.978048    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="kube-rbac-proxy-thanos"
Aug 04 18:02:35.978145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.978096    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="prometheus"
Aug 04 18:02:35.978145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.978102    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="kube-rbac-proxy-thanos"
Aug 04 18:02:35.978145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.978111    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="thanos-sidecar"
Aug 04 18:02:35.978145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.978119    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="kube-rbac-proxy"
Aug 04 18:02:35.978145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.978127    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="kube-rbac-proxy-web"
Aug 04 18:02:35.978145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.978132    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" containerName="config-reloader"
Aug 04 18:02:35.981893 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.981877    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:35.985778 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.985755    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Aug 04 18:02:35.985900 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.985793    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Aug 04 18:02:35.985900 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.985813    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Aug 04 18:02:35.986130 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.986115    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zj94x\""
Aug 04 18:02:35.986247 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.986215    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Aug 04 18:02:35.986623 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.986609    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Aug 04 18:02:35.986762 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.986743    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Aug 04 18:02:35.986762 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.986745    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Aug 04 18:02:35.987053 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.987038    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Aug 04 18:02:35.987494 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.987279    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e4kbmrl6ugl4\""
Aug 04 18:02:35.987494 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.987317    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Aug 04 18:02:35.987494 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.987423    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Aug 04 18:02:35.988006 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.987985    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Aug 04 18:02:35.990651 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.990519    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Aug 04 18:02:35.992016 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.991998    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Aug 04 18:02:35.994748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:35.994731    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Aug 04 18:02:36.005323 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.005299    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:02:36.113642 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.113606    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.113642 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.113639    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.113828 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.113662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.113828 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.113680    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjl97\" (UniqueName: \"kubernetes.io/projected/af261892-1e1c-4d98-9aaf-5a292f5c78f5-kube-api-access-fjl97\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.113828 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.113731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.113828 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.113801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.113828 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.113826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.114000 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.113852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af261892-1e1c-4d98-9aaf-5a292f5c78f5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.114000 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.113877    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af261892-1e1c-4d98-9aaf-5a292f5c78f5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.114000 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.113893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af261892-1e1c-4d98-9aaf-5a292f5c78f5-config-out\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.114000 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.113939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.114000 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.113963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.114000 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.113980    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-config\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.114000 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.113998    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.114243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.114014    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.114243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.114031    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.114243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.114052    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.114243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.114069    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.114243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.114121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-web-config\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.114243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.114169    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.214962 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.214913    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.214962 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.214963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-web-config\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.214984    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215024    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjl97\" (UniqueName: \"kubernetes.io/projected/af261892-1e1c-4d98-9aaf-5a292f5c78f5-kube-api-access-fjl97\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215110    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215192    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af261892-1e1c-4d98-9aaf-5a292f5c78f5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215227    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af261892-1e1c-4d98-9aaf-5a292f5c78f5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215255    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af261892-1e1c-4d98-9aaf-5a292f5c78f5-config-out\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215342    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-config\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215399    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215454    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.215685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.216082 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215822    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.216082 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.215822    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.217203 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.216739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.217203 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.216838    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.217203 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.217074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.217677 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.217649    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.218663 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.218494    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af261892-1e1c-4d98-9aaf-5a292f5c78f5-config-out\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.220387 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.220035    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.220495 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.220471    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-config\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.220557 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.220477    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af261892-1e1c-4d98-9aaf-5a292f5c78f5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.221143 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.220739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/af261892-1e1c-4d98-9aaf-5a292f5c78f5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.221143 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.220768    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-web-config\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.221143 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.221038    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.221143 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.221094    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.225811 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.221469    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.225811 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.221635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/af261892-1e1c-4d98-9aaf-5a292f5c78f5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.225811 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.221695    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.225811 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.223461    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.226047 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.225869    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/af261892-1e1c-4d98-9aaf-5a292f5c78f5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.226297 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.226272    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjl97\" (UniqueName: \"kubernetes.io/projected/af261892-1e1c-4d98-9aaf-5a292f5c78f5-kube-api-access-fjl97\") pod \"prometheus-k8s-0\" (UID: \"af261892-1e1c-4d98-9aaf-5a292f5c78f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.294825 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.294731    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:02:36.438209 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.438182    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Aug 04 18:02:36.440250 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:02:36.440226    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf261892_1e1c_4d98_9aaf_5a292f5c78f5.slice/crio-672a90a44a3234fc1b5e4f4333c0ef61501a7a6ab2cef289910274ec4acb303e WatchSource:0}: Error finding container 672a90a44a3234fc1b5e4f4333c0ef61501a7a6ab2cef289910274ec4acb303e: Status 404 returned error can't find the container with id 672a90a44a3234fc1b5e4f4333c0ef61501a7a6ab2cef289910274ec4acb303e
Aug 04 18:02:36.915989 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.915943    2566 generic.go:356] "Generic (PLEG): container finished" podID="af261892-1e1c-4d98-9aaf-5a292f5c78f5" containerID="322484a5bbe3dc59eb9526fcb88114b972296bf6ecb416b5487f253b6a7f590e" exitCode=0
Aug 04 18:02:36.916378 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.916003    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af261892-1e1c-4d98-9aaf-5a292f5c78f5","Type":"ContainerDied","Data":"322484a5bbe3dc59eb9526fcb88114b972296bf6ecb416b5487f253b6a7f590e"}
Aug 04 18:02:36.916378 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:36.916068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af261892-1e1c-4d98-9aaf-5a292f5c78f5","Type":"ContainerStarted","Data":"672a90a44a3234fc1b5e4f4333c0ef61501a7a6ab2cef289910274ec4acb303e"}
Aug 04 18:02:37.195716 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:37.195672    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f63bfef0-7395-4857-9f3b-a59161a9703b" path="/var/lib/kubelet/pods/f63bfef0-7395-4857-9f3b-a59161a9703b/volumes"
Aug 04 18:02:37.923022 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:37.922983    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af261892-1e1c-4d98-9aaf-5a292f5c78f5","Type":"ContainerStarted","Data":"600adccb695cb5c05997bce9563422aa3618109052aea53b07810d7f8d2ef02d"}
Aug 04 18:02:37.923408 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:37.923067    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af261892-1e1c-4d98-9aaf-5a292f5c78f5","Type":"ContainerStarted","Data":"1d71a18ba80e15a444c7588f450e80cd8258a2b691191c8c5b7ca515b3ea7c68"}
Aug 04 18:02:37.923408 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:37.923107    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af261892-1e1c-4d98-9aaf-5a292f5c78f5","Type":"ContainerStarted","Data":"586a6674f6769ea3ee5508e59527c57807a878b315d16966175833f841645435"}
Aug 04 18:02:37.923408 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:37.923149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af261892-1e1c-4d98-9aaf-5a292f5c78f5","Type":"ContainerStarted","Data":"8c4ee35ef29a988f24f8ce748d39b44269f6eae877b855692afbbad92cd9ed28"}
Aug 04 18:02:37.923408 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:37.923184    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af261892-1e1c-4d98-9aaf-5a292f5c78f5","Type":"ContainerStarted","Data":"3cf70fff81f6557e37ecb3c3fe273135311457bc6172df60342f4ff027086e3e"}
Aug 04 18:02:37.923408 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:37.923217    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"af261892-1e1c-4d98-9aaf-5a292f5c78f5","Type":"ContainerStarted","Data":"8b0770084f615b663b44de9060e9b006b00d40571d8cdf399331d0c60103210c"}
Aug 04 18:02:37.956094 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:37.956033    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.9345730899999998 podStartE2EDuration="2.956009985s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.436895ms" isStatefulPod=false podCreationTimestamp="2026-08-04 18:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:37.954394203 +0000 UTC m=+157.245008375" watchObservedRunningTime="2026-08-04 18:02:37.956009985 +0000 UTC m=+157.246624157"
Aug 04 18:02:41.295769 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:02:41.295719    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:03:36.295791 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:03:36.295754    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:03:36.311332 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:03:36.311307    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:03:37.175866 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:03:37.175831    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Aug 04 18:05:01.100125 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:05:01.100091    2566 kubelet.go:1648] "Image garbage collection succeeded"
Aug 04 18:08:44.449395 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:44.449349    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-8588677cfb-r2pkl"]
Aug 04 18:08:44.452711 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:44.452692    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-r2pkl"
Aug 04 18:08:44.454976 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:44.454920    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Aug 04 18:08:44.455319 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:44.455303    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Aug 04 18:08:44.456116 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:44.456095    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Aug 04 18:08:44.456205 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:44.456095    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Aug 04 18:08:44.456205 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:44.456164    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-l7bql\""
Aug 04 18:08:44.463519 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:44.463500    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-r2pkl"]
Aug 04 18:08:44.551896 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:44.551861    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4b2w\" (UniqueName: \"kubernetes.io/projected/dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3-kube-api-access-t4b2w\") pod \"odh-model-controller-8588677cfb-r2pkl\" (UID: \"dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3\") " pod="kserve/odh-model-controller-8588677cfb-r2pkl"
Aug 04 18:08:44.551896 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:44.551898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3-metrics-certs\") pod \"odh-model-controller-8588677cfb-r2pkl\" (UID: \"dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3\") " pod="kserve/odh-model-controller-8588677cfb-r2pkl"
Aug 04 18:08:44.552129 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:44.551926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3-cert\") pod \"odh-model-controller-8588677cfb-r2pkl\" (UID: \"dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3\") " pod="kserve/odh-model-controller-8588677cfb-r2pkl"
Aug 04 18:08:44.652476 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:44.652439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4b2w\" (UniqueName: \"kubernetes.io/projected/dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3-kube-api-access-t4b2w\") pod \"odh-model-controller-8588677cfb-r2pkl\" (UID: \"dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3\") " pod="kserve/odh-model-controller-8588677cfb-r2pkl"
Aug 04 18:08:44.652476 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:44.652484    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3-metrics-certs\") pod \"odh-model-controller-8588677cfb-r2pkl\" (UID: \"dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3\") " pod="kserve/odh-model-controller-8588677cfb-r2pkl"
Aug 04 18:08:44.652672 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:08:44.652579    2566 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Aug 04 18:08:44.652672 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:44.652602    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3-cert\") pod \"odh-model-controller-8588677cfb-r2pkl\" (UID: \"dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3\") " pod="kserve/odh-model-controller-8588677cfb-r2pkl"
Aug 04 18:08:44.652672 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:08:44.652633    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3-metrics-certs podName:dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3 nodeName:}" failed. No retries permitted until 2026-08-04 18:08:45.152613156 +0000 UTC m=+524.443227323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3-metrics-certs") pod "odh-model-controller-8588677cfb-r2pkl" (UID: "dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3") : secret "odh-model-controller-metrics-tls" not found
Aug 04 18:08:44.652870 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:08:44.652679    2566 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Aug 04 18:08:44.652870 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:08:44.652735    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3-cert podName:dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3 nodeName:}" failed. No retries permitted until 2026-08-04 18:08:45.152720985 +0000 UTC m=+524.443335134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3-cert") pod "odh-model-controller-8588677cfb-r2pkl" (UID: "dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3") : secret "odh-model-controller-webhook-cert" not found
Aug 04 18:08:44.661553 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:44.661522    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4b2w\" (UniqueName: \"kubernetes.io/projected/dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3-kube-api-access-t4b2w\") pod \"odh-model-controller-8588677cfb-r2pkl\" (UID: \"dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3\") " pod="kserve/odh-model-controller-8588677cfb-r2pkl"
Aug 04 18:08:45.157980 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:45.157939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3-metrics-certs\") pod \"odh-model-controller-8588677cfb-r2pkl\" (UID: \"dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3\") " pod="kserve/odh-model-controller-8588677cfb-r2pkl"
Aug 04 18:08:45.158187 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:45.157990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3-cert\") pod \"odh-model-controller-8588677cfb-r2pkl\" (UID: \"dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3\") " pod="kserve/odh-model-controller-8588677cfb-r2pkl"
Aug 04 18:08:45.160592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:45.160566    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3-cert\") pod \"odh-model-controller-8588677cfb-r2pkl\" (UID: \"dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3\") " pod="kserve/odh-model-controller-8588677cfb-r2pkl"
Aug 04 18:08:45.160592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:45.160591    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3-metrics-certs\") pod \"odh-model-controller-8588677cfb-r2pkl\" (UID: \"dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3\") " pod="kserve/odh-model-controller-8588677cfb-r2pkl"
Aug 04 18:08:45.363649 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:45.363611    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-r2pkl"
Aug 04 18:08:45.493093 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:45.493067    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-r2pkl"]
Aug 04 18:08:45.495178 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:08:45.495153    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6b2f08_4c9e_44a0_a1da_4e63bfb92dc3.slice/crio-af4f4ccf638894ef0acff8c4b957360eb883c92ef3ffd2694318afec0f53cf40 WatchSource:0}: Error finding container af4f4ccf638894ef0acff8c4b957360eb883c92ef3ffd2694318afec0f53cf40: Status 404 returned error can't find the container with id af4f4ccf638894ef0acff8c4b957360eb883c92ef3ffd2694318afec0f53cf40
Aug 04 18:08:45.496416 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:45.496401    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:08:46.399116 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:46.399028    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-r2pkl" event={"ID":"dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3","Type":"ContainerStarted","Data":"af4f4ccf638894ef0acff8c4b957360eb883c92ef3ffd2694318afec0f53cf40"}
Aug 04 18:08:48.409325 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:48.409288    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-r2pkl" event={"ID":"dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3","Type":"ContainerStarted","Data":"11a238b7fb1c0afe65c02cca998da6318cb07d04d57563b726e2671d30acb396"}
Aug 04 18:08:48.409723 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:48.409395    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-8588677cfb-r2pkl"
Aug 04 18:08:48.428472 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:48.428421    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-8588677cfb-r2pkl" podStartSLOduration=1.959578165 podStartE2EDuration="4.428404328s" totalImagesPullingTime="2.468826163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:08:48.427943531 +0000 UTC m=+527.718557725" watchObservedRunningTime="2026-08-04 18:08:48.428404328 +0000 UTC m=+527.719018498"
Aug 04 18:08:59.415303 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:08:59.415227    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-8588677cfb-r2pkl"
Aug 04 18:09:00.267371 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:00.267324    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-rtpqq"]
Aug 04 18:09:00.276412 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:00.276380    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rtpqq"
Aug 04 18:09:00.279228 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:00.279207    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Aug 04 18:09:00.279388 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:00.279282    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-gqc29\""
Aug 04 18:09:00.280237 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:00.280213    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-rtpqq"]
Aug 04 18:09:00.394406 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:00.394348    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93aeb3ba-444e-4165-901a-cd2512c2d6dc-model-cache\") pod \"s3-init-rtpqq\" (UID: \"93aeb3ba-444e-4165-901a-cd2512c2d6dc\") " pod="kserve/s3-init-rtpqq"
Aug 04 18:09:00.394584 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:00.394418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgc2q\" (UniqueName: \"kubernetes.io/projected/93aeb3ba-444e-4165-901a-cd2512c2d6dc-kube-api-access-dgc2q\") pod \"s3-init-rtpqq\" (UID: \"93aeb3ba-444e-4165-901a-cd2512c2d6dc\") " pod="kserve/s3-init-rtpqq"
Aug 04 18:09:00.495334 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:00.495301    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93aeb3ba-444e-4165-901a-cd2512c2d6dc-model-cache\") pod \"s3-init-rtpqq\" (UID: \"93aeb3ba-444e-4165-901a-cd2512c2d6dc\") " pod="kserve/s3-init-rtpqq"
Aug 04 18:09:00.495819 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:00.495345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgc2q\" (UniqueName: \"kubernetes.io/projected/93aeb3ba-444e-4165-901a-cd2512c2d6dc-kube-api-access-dgc2q\") pod \"s3-init-rtpqq\" (UID: \"93aeb3ba-444e-4165-901a-cd2512c2d6dc\") " pod="kserve/s3-init-rtpqq"
Aug 04 18:09:00.495819 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:00.495753    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93aeb3ba-444e-4165-901a-cd2512c2d6dc-model-cache\") pod \"s3-init-rtpqq\" (UID: \"93aeb3ba-444e-4165-901a-cd2512c2d6dc\") " pod="kserve/s3-init-rtpqq"
Aug 04 18:09:00.504856 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:00.504832    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgc2q\" (UniqueName: \"kubernetes.io/projected/93aeb3ba-444e-4165-901a-cd2512c2d6dc-kube-api-access-dgc2q\") pod \"s3-init-rtpqq\" (UID: \"93aeb3ba-444e-4165-901a-cd2512c2d6dc\") " pod="kserve/s3-init-rtpqq"
Aug 04 18:09:00.599645 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:00.599608    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rtpqq"
Aug 04 18:09:00.731693 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:00.731669    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-rtpqq"]
Aug 04 18:09:00.733988 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:09:00.733965    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93aeb3ba_444e_4165_901a_cd2512c2d6dc.slice/crio-26aafbbe16de23c7ac4e5d99b8e0dc33cb5dd693ac5bccaf789959b38f60fe96 WatchSource:0}: Error finding container 26aafbbe16de23c7ac4e5d99b8e0dc33cb5dd693ac5bccaf789959b38f60fe96: Status 404 returned error can't find the container with id 26aafbbe16de23c7ac4e5d99b8e0dc33cb5dd693ac5bccaf789959b38f60fe96
Aug 04 18:09:01.460701 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:01.460665    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rtpqq" event={"ID":"93aeb3ba-444e-4165-901a-cd2512c2d6dc","Type":"ContainerStarted","Data":"26aafbbe16de23c7ac4e5d99b8e0dc33cb5dd693ac5bccaf789959b38f60fe96"}
Aug 04 18:09:05.479705 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:05.479665    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rtpqq" event={"ID":"93aeb3ba-444e-4165-901a-cd2512c2d6dc","Type":"ContainerStarted","Data":"e2b568ea92efbec46ddb6e7ceae6bdf2fb144048481062c9986386a2a32c9d1b"}
Aug 04 18:09:07.489392 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:07.489343    2566 generic.go:356] "Generic (PLEG): container finished" podID="93aeb3ba-444e-4165-901a-cd2512c2d6dc" containerID="e2b568ea92efbec46ddb6e7ceae6bdf2fb144048481062c9986386a2a32c9d1b" exitCode=0
Aug 04 18:09:07.489746 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:07.489417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rtpqq" event={"ID":"93aeb3ba-444e-4165-901a-cd2512c2d6dc","Type":"ContainerDied","Data":"e2b568ea92efbec46ddb6e7ceae6bdf2fb144048481062c9986386a2a32c9d1b"}
Aug 04 18:09:11.820645 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:11.820620    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Aug 04 18:09:12.515664 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:12.515619    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rtpqq" event={"ID":"93aeb3ba-444e-4165-901a-cd2512c2d6dc","Type":"ContainerStarted","Data":"76dbe34331232b575d5fb9166999412e3d58cc1831690d6f0fcbe2b504657c75"}
Aug 04 18:09:12.533724 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:09:12.533671    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-rtpqq" podStartSLOduration=1.741630837 podStartE2EDuration="12.533657128s" totalImagesPullingTime="8.624805842s" totalInitContainerRuntime="2.167220449s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:09:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:09:12.532496752 +0000 UTC m=+551.823110925" watchObservedRunningTime="2026-08-04 18:09:12.533657128 +0000 UTC m=+551.824271299"
Aug 04 18:10:17.788374 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:17.788317    2566 generic.go:356] "Generic (PLEG): container finished" podID="93aeb3ba-444e-4165-901a-cd2512c2d6dc" containerID="76dbe34331232b575d5fb9166999412e3d58cc1831690d6f0fcbe2b504657c75" exitCode=0
Aug 04 18:10:17.788776 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:17.788393    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rtpqq" event={"ID":"93aeb3ba-444e-4165-901a-cd2512c2d6dc","Type":"ContainerDied","Data":"76dbe34331232b575d5fb9166999412e3d58cc1831690d6f0fcbe2b504657c75"}
Aug 04 18:10:18.919040 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:18.919016    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rtpqq"
Aug 04 18:10:19.084687 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:19.084655    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93aeb3ba-444e-4165-901a-cd2512c2d6dc-model-cache\") pod \"93aeb3ba-444e-4165-901a-cd2512c2d6dc\" (UID: \"93aeb3ba-444e-4165-901a-cd2512c2d6dc\") "
Aug 04 18:10:19.084897 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:19.084731    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgc2q\" (UniqueName: \"kubernetes.io/projected/93aeb3ba-444e-4165-901a-cd2512c2d6dc-kube-api-access-dgc2q\") pod \"93aeb3ba-444e-4165-901a-cd2512c2d6dc\" (UID: \"93aeb3ba-444e-4165-901a-cd2512c2d6dc\") "
Aug 04 18:10:19.086874 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:19.086844    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93aeb3ba-444e-4165-901a-cd2512c2d6dc-kube-api-access-dgc2q" (OuterVolumeSpecName: "kube-api-access-dgc2q") pod "93aeb3ba-444e-4165-901a-cd2512c2d6dc" (UID: "93aeb3ba-444e-4165-901a-cd2512c2d6dc"). InnerVolumeSpecName "kube-api-access-dgc2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:10:19.103885 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:19.103854    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93aeb3ba-444e-4165-901a-cd2512c2d6dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "93aeb3ba-444e-4165-901a-cd2512c2d6dc" (UID: "93aeb3ba-444e-4165-901a-cd2512c2d6dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:10:19.185611 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:19.185579    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93aeb3ba-444e-4165-901a-cd2512c2d6dc-model-cache\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:10:19.185611 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:19.185610    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgc2q\" (UniqueName: \"kubernetes.io/projected/93aeb3ba-444e-4165-901a-cd2512c2d6dc-kube-api-access-dgc2q\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:10:19.797765 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:19.797726    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rtpqq" event={"ID":"93aeb3ba-444e-4165-901a-cd2512c2d6dc","Type":"ContainerDied","Data":"26aafbbe16de23c7ac4e5d99b8e0dc33cb5dd693ac5bccaf789959b38f60fe96"}
Aug 04 18:10:19.797765 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:19.797743    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rtpqq"
Aug 04 18:10:19.797765 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:19.797765    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26aafbbe16de23c7ac4e5d99b8e0dc33cb5dd693ac5bccaf789959b38f60fe96"
Aug 04 18:10:20.560038 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.560002    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7"]
Aug 04 18:10:20.560447 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.560335    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93aeb3ba-444e-4165-901a-cd2512c2d6dc" containerName="download-hf-model"
Aug 04 18:10:20.560447 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.560347    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="93aeb3ba-444e-4165-901a-cd2512c2d6dc" containerName="download-hf-model"
Aug 04 18:10:20.560447 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.560374    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93aeb3ba-444e-4165-901a-cd2512c2d6dc" containerName="s3-init"
Aug 04 18:10:20.560447 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.560380    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="93aeb3ba-444e-4165-901a-cd2512c2d6dc" containerName="s3-init"
Aug 04 18:10:20.560447 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.560442    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="93aeb3ba-444e-4165-901a-cd2512c2d6dc" containerName="s3-init"
Aug 04 18:10:20.563519 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.563501    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7"
Aug 04 18:10:20.566066 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.566046    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-gqc29\""
Aug 04 18:10:20.566066 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.566065    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Aug 04 18:10:20.570943 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.570921    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7"]
Aug 04 18:10:20.700774 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.700734    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tpd2\" (UniqueName: \"kubernetes.io/projected/2386e73f-398b-4e0d-9715-7d3b272fb269-kube-api-access-8tpd2\") pod \"seaweedfs-tls-custom-6644bf5d6c-2pzq7\" (UID: \"2386e73f-398b-4e0d-9715-7d3b272fb269\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7"
Aug 04 18:10:20.700944 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.700786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2386e73f-398b-4e0d-9715-7d3b272fb269-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-2pzq7\" (UID: \"2386e73f-398b-4e0d-9715-7d3b272fb269\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7"
Aug 04 18:10:20.801487 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.801457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tpd2\" (UniqueName: \"kubernetes.io/projected/2386e73f-398b-4e0d-9715-7d3b272fb269-kube-api-access-8tpd2\") pod \"seaweedfs-tls-custom-6644bf5d6c-2pzq7\" (UID: \"2386e73f-398b-4e0d-9715-7d3b272fb269\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7"
Aug 04 18:10:20.801631 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.801495    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2386e73f-398b-4e0d-9715-7d3b272fb269-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-2pzq7\" (UID: \"2386e73f-398b-4e0d-9715-7d3b272fb269\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7"
Aug 04 18:10:20.801861 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.801846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2386e73f-398b-4e0d-9715-7d3b272fb269-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-2pzq7\" (UID: \"2386e73f-398b-4e0d-9715-7d3b272fb269\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7"
Aug 04 18:10:20.811782 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.811715    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tpd2\" (UniqueName: \"kubernetes.io/projected/2386e73f-398b-4e0d-9715-7d3b272fb269-kube-api-access-8tpd2\") pod \"seaweedfs-tls-custom-6644bf5d6c-2pzq7\" (UID: \"2386e73f-398b-4e0d-9715-7d3b272fb269\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7"
Aug 04 18:10:20.873214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:20.873181    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7"
Aug 04 18:10:21.010630 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:21.010607    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7"]
Aug 04 18:10:21.012884 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:10:21.012846    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2386e73f_398b_4e0d_9715_7d3b272fb269.slice/crio-e4bd2151e91920d69ebad44fa23275f2c023703e5ced9bf4d97ac210784aeb32 WatchSource:0}: Error finding container e4bd2151e91920d69ebad44fa23275f2c023703e5ced9bf4d97ac210784aeb32: Status 404 returned error can't find the container with id e4bd2151e91920d69ebad44fa23275f2c023703e5ced9bf4d97ac210784aeb32
Aug 04 18:10:21.807181 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:21.807144    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7" event={"ID":"2386e73f-398b-4e0d-9715-7d3b272fb269","Type":"ContainerStarted","Data":"e4bd2151e91920d69ebad44fa23275f2c023703e5ced9bf4d97ac210784aeb32"}
Aug 04 18:10:24.820704 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:24.820669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7" event={"ID":"2386e73f-398b-4e0d-9715-7d3b272fb269","Type":"ContainerStarted","Data":"0b40a9155aed45972e132e3f5aa73d7188cb5e742d780884cd39bf3336cd09bf"}
Aug 04 18:10:24.838166 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:24.838120    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7" podStartSLOduration=1.316118205 podStartE2EDuration="4.838104581s" totalImagesPullingTime="3.521986376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:24.83779777 +0000 UTC m=+624.128411970" watchObservedRunningTime="2026-08-04 18:10:24.838104581 +0000 UTC m=+624.128718752"
Aug 04 18:10:26.787716 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:26.787683    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7"]
Aug 04 18:10:26.829479 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:26.829438    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7" podUID="2386e73f-398b-4e0d-9715-7d3b272fb269" containerName="seaweedfs-tls-custom" containerID="cri-o://0b40a9155aed45972e132e3f5aa73d7188cb5e742d780884cd39bf3336cd09bf" gracePeriod=30
Aug 04 18:10:28.070065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.070041    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7"
Aug 04 18:10:28.169083 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.168998    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tpd2\" (UniqueName: \"kubernetes.io/projected/2386e73f-398b-4e0d-9715-7d3b272fb269-kube-api-access-8tpd2\") pod \"2386e73f-398b-4e0d-9715-7d3b272fb269\" (UID: \"2386e73f-398b-4e0d-9715-7d3b272fb269\") "
Aug 04 18:10:28.169216 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.169197    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2386e73f-398b-4e0d-9715-7d3b272fb269-data\") pod \"2386e73f-398b-4e0d-9715-7d3b272fb269\" (UID: \"2386e73f-398b-4e0d-9715-7d3b272fb269\") "
Aug 04 18:10:28.170518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.170485    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2386e73f-398b-4e0d-9715-7d3b272fb269-data" (OuterVolumeSpecName: "data") pod "2386e73f-398b-4e0d-9715-7d3b272fb269" (UID: "2386e73f-398b-4e0d-9715-7d3b272fb269"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:10:28.171090 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.171059    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2386e73f-398b-4e0d-9715-7d3b272fb269-kube-api-access-8tpd2" (OuterVolumeSpecName: "kube-api-access-8tpd2") pod "2386e73f-398b-4e0d-9715-7d3b272fb269" (UID: "2386e73f-398b-4e0d-9715-7d3b272fb269"). InnerVolumeSpecName "kube-api-access-8tpd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:10:28.270866 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.270813    2566 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2386e73f-398b-4e0d-9715-7d3b272fb269-data\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:10:28.270866 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.270862    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tpd2\" (UniqueName: \"kubernetes.io/projected/2386e73f-398b-4e0d-9715-7d3b272fb269-kube-api-access-8tpd2\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:10:28.838730 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.838691    2566 generic.go:356] "Generic (PLEG): container finished" podID="2386e73f-398b-4e0d-9715-7d3b272fb269" containerID="0b40a9155aed45972e132e3f5aa73d7188cb5e742d780884cd39bf3336cd09bf" exitCode=0
Aug 04 18:10:28.838730 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.838732    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7"
Aug 04 18:10:28.838939 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.838774    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7" event={"ID":"2386e73f-398b-4e0d-9715-7d3b272fb269","Type":"ContainerDied","Data":"0b40a9155aed45972e132e3f5aa73d7188cb5e742d780884cd39bf3336cd09bf"}
Aug 04 18:10:28.838939 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.838828    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7" event={"ID":"2386e73f-398b-4e0d-9715-7d3b272fb269","Type":"ContainerDied","Data":"e4bd2151e91920d69ebad44fa23275f2c023703e5ced9bf4d97ac210784aeb32"}
Aug 04 18:10:28.838939 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.838850    2566 scope.go:117] "RemoveContainer" containerID="0b40a9155aed45972e132e3f5aa73d7188cb5e742d780884cd39bf3336cd09bf"
Aug 04 18:10:28.848870 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.848855    2566 scope.go:117] "RemoveContainer" containerID="0b40a9155aed45972e132e3f5aa73d7188cb5e742d780884cd39bf3336cd09bf"
Aug 04 18:10:28.849130 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:10:28.849111    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b40a9155aed45972e132e3f5aa73d7188cb5e742d780884cd39bf3336cd09bf\": container with ID starting with 0b40a9155aed45972e132e3f5aa73d7188cb5e742d780884cd39bf3336cd09bf not found: ID does not exist" containerID="0b40a9155aed45972e132e3f5aa73d7188cb5e742d780884cd39bf3336cd09bf"
Aug 04 18:10:28.849178 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.849143    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b40a9155aed45972e132e3f5aa73d7188cb5e742d780884cd39bf3336cd09bf"} err="failed to get container status \"0b40a9155aed45972e132e3f5aa73d7188cb5e742d780884cd39bf3336cd09bf\": rpc error: code = NotFound desc = could not find container \"0b40a9155aed45972e132e3f5aa73d7188cb5e742d780884cd39bf3336cd09bf\": container with ID starting with 0b40a9155aed45972e132e3f5aa73d7188cb5e742d780884cd39bf3336cd09bf not found: ID does not exist"
Aug 04 18:10:28.862323 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.862296    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7"]
Aug 04 18:10:28.868788 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.868767    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-2pzq7"]
Aug 04 18:10:28.900214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.900179    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-kkq9j"]
Aug 04 18:10:28.900600 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.900584    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2386e73f-398b-4e0d-9715-7d3b272fb269" containerName="seaweedfs-tls-custom"
Aug 04 18:10:28.900696 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.900603    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2386e73f-398b-4e0d-9715-7d3b272fb269" containerName="seaweedfs-tls-custom"
Aug 04 18:10:28.900750 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.900704    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2386e73f-398b-4e0d-9715-7d3b272fb269" containerName="seaweedfs-tls-custom"
Aug 04 18:10:28.904865 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.904845    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-kkq9j"
Aug 04 18:10:28.911328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.907852    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-gqc29\""
Aug 04 18:10:28.911328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.907865    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Aug 04 18:10:28.911328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.907932    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Aug 04 18:10:28.913743 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:28.913710    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-kkq9j"]
Aug 04 18:10:29.078943 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:29.078902    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75hx\" (UniqueName: \"kubernetes.io/projected/67d8f728-7658-4e60-852d-c879cf1cc59c-kube-api-access-q75hx\") pod \"seaweedfs-tls-custom-8b44d767d-kkq9j\" (UID: \"67d8f728-7658-4e60-852d-c879cf1cc59c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kkq9j"
Aug 04 18:10:29.079311 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:29.078974    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/67d8f728-7658-4e60-852d-c879cf1cc59c-data\") pod \"seaweedfs-tls-custom-8b44d767d-kkq9j\" (UID: \"67d8f728-7658-4e60-852d-c879cf1cc59c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kkq9j"
Aug 04 18:10:29.079311 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:29.079007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/67d8f728-7658-4e60-852d-c879cf1cc59c-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-kkq9j\" (UID: \"67d8f728-7658-4e60-852d-c879cf1cc59c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kkq9j"
Aug 04 18:10:29.179836 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:29.179766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/67d8f728-7658-4e60-852d-c879cf1cc59c-data\") pod \"seaweedfs-tls-custom-8b44d767d-kkq9j\" (UID: \"67d8f728-7658-4e60-852d-c879cf1cc59c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kkq9j"
Aug 04 18:10:29.179836 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:29.179807    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/67d8f728-7658-4e60-852d-c879cf1cc59c-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-kkq9j\" (UID: \"67d8f728-7658-4e60-852d-c879cf1cc59c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kkq9j"
Aug 04 18:10:29.179993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:29.179866    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75hx\" (UniqueName: \"kubernetes.io/projected/67d8f728-7658-4e60-852d-c879cf1cc59c-kube-api-access-q75hx\") pod \"seaweedfs-tls-custom-8b44d767d-kkq9j\" (UID: \"67d8f728-7658-4e60-852d-c879cf1cc59c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kkq9j"
Aug 04 18:10:29.180147 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:29.180126    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/67d8f728-7658-4e60-852d-c879cf1cc59c-data\") pod \"seaweedfs-tls-custom-8b44d767d-kkq9j\" (UID: \"67d8f728-7658-4e60-852d-c879cf1cc59c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kkq9j"
Aug 04 18:10:29.182199 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:29.182167    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/67d8f728-7658-4e60-852d-c879cf1cc59c-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-kkq9j\" (UID: \"67d8f728-7658-4e60-852d-c879cf1cc59c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kkq9j"
Aug 04 18:10:29.189828 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:29.189808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75hx\" (UniqueName: \"kubernetes.io/projected/67d8f728-7658-4e60-852d-c879cf1cc59c-kube-api-access-q75hx\") pod \"seaweedfs-tls-custom-8b44d767d-kkq9j\" (UID: \"67d8f728-7658-4e60-852d-c879cf1cc59c\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kkq9j"
Aug 04 18:10:29.195263 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:29.195234    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2386e73f-398b-4e0d-9715-7d3b272fb269" path="/var/lib/kubelet/pods/2386e73f-398b-4e0d-9715-7d3b272fb269/volumes"
Aug 04 18:10:29.219865 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:29.219834    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-kkq9j"
Aug 04 18:10:29.347615 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:29.347584    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-kkq9j"]
Aug 04 18:10:29.350510 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:10:29.350484    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d8f728_7658_4e60_852d_c879cf1cc59c.slice/crio-1e31b81951569877d86a5ebd4568fd45156e982f96087abe631baa869e3fe318 WatchSource:0}: Error finding container 1e31b81951569877d86a5ebd4568fd45156e982f96087abe631baa869e3fe318: Status 404 returned error can't find the container with id 1e31b81951569877d86a5ebd4568fd45156e982f96087abe631baa869e3fe318
Aug 04 18:10:29.846329 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:29.846298    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-kkq9j" event={"ID":"67d8f728-7658-4e60-852d-c879cf1cc59c","Type":"ContainerStarted","Data":"e5a5231c5a65b46f215c1f0bb37d141b5b09e96c04d195519186ac8f9079f598"}
Aug 04 18:10:29.846329 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:29.846345    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-kkq9j" event={"ID":"67d8f728-7658-4e60-852d-c879cf1cc59c","Type":"ContainerStarted","Data":"1e31b81951569877d86a5ebd4568fd45156e982f96087abe631baa869e3fe318"}
Aug 04 18:10:29.864680 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:29.864632    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-kkq9j" podStartSLOduration=1.575025549 podStartE2EDuration="1.86461635s" totalImagesPullingTime="289.590801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:29.864241637 +0000 UTC m=+629.154855809" watchObservedRunningTime="2026-08-04 18:10:29.86461635 +0000 UTC m=+629.155230549"
Aug 04 18:10:30.175521 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:30.175481    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-s6qrd"]
Aug 04 18:10:30.178776 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:30.178758    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-s6qrd"
Aug 04 18:10:30.188627 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:30.188599    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-s6qrd"]
Aug 04 18:10:30.286578 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:30.286544    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52b71d8a-53dd-4b07-9cbd-6b08ee2343c3-model-cache\") pod \"s3-tls-init-custom-s6qrd\" (UID: \"52b71d8a-53dd-4b07-9cbd-6b08ee2343c3\") " pod="kserve/s3-tls-init-custom-s6qrd"
Aug 04 18:10:30.286786 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:30.286665    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw65x\" (UniqueName: \"kubernetes.io/projected/52b71d8a-53dd-4b07-9cbd-6b08ee2343c3-kube-api-access-tw65x\") pod \"s3-tls-init-custom-s6qrd\" (UID: \"52b71d8a-53dd-4b07-9cbd-6b08ee2343c3\") " pod="kserve/s3-tls-init-custom-s6qrd"
Aug 04 18:10:30.387807 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:30.387771    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52b71d8a-53dd-4b07-9cbd-6b08ee2343c3-model-cache\") pod \"s3-tls-init-custom-s6qrd\" (UID: \"52b71d8a-53dd-4b07-9cbd-6b08ee2343c3\") " pod="kserve/s3-tls-init-custom-s6qrd"
Aug 04 18:10:30.387989 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:30.387839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw65x\" (UniqueName: \"kubernetes.io/projected/52b71d8a-53dd-4b07-9cbd-6b08ee2343c3-kube-api-access-tw65x\") pod \"s3-tls-init-custom-s6qrd\" (UID: \"52b71d8a-53dd-4b07-9cbd-6b08ee2343c3\") " pod="kserve/s3-tls-init-custom-s6qrd"
Aug 04 18:10:30.388165 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:30.388145    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52b71d8a-53dd-4b07-9cbd-6b08ee2343c3-model-cache\") pod \"s3-tls-init-custom-s6qrd\" (UID: \"52b71d8a-53dd-4b07-9cbd-6b08ee2343c3\") " pod="kserve/s3-tls-init-custom-s6qrd"
Aug 04 18:10:30.398086 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:30.398058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw65x\" (UniqueName: \"kubernetes.io/projected/52b71d8a-53dd-4b07-9cbd-6b08ee2343c3-kube-api-access-tw65x\") pod \"s3-tls-init-custom-s6qrd\" (UID: \"52b71d8a-53dd-4b07-9cbd-6b08ee2343c3\") " pod="kserve/s3-tls-init-custom-s6qrd"
Aug 04 18:10:30.501004 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:30.500915    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-s6qrd"
Aug 04 18:10:30.629897 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:30.629730    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-s6qrd"]
Aug 04 18:10:30.632525 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:10:30.632487    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52b71d8a_53dd_4b07_9cbd_6b08ee2343c3.slice/crio-c83adf0454afa51b1346a47e75e42f85f88acda742289d907455d7e73551bb49 WatchSource:0}: Error finding container c83adf0454afa51b1346a47e75e42f85f88acda742289d907455d7e73551bb49: Status 404 returned error can't find the container with id c83adf0454afa51b1346a47e75e42f85f88acda742289d907455d7e73551bb49
Aug 04 18:10:30.852185 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:30.852149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s6qrd" event={"ID":"52b71d8a-53dd-4b07-9cbd-6b08ee2343c3","Type":"ContainerStarted","Data":"76a7b3c469d5bb699aeb4ee3d625b6082479944b4f9276e3c1385b30248a8ecf"}
Aug 04 18:10:30.852552 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:30.852197    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s6qrd" event={"ID":"52b71d8a-53dd-4b07-9cbd-6b08ee2343c3","Type":"ContainerStarted","Data":"c83adf0454afa51b1346a47e75e42f85f88acda742289d907455d7e73551bb49"}
Aug 04 18:10:32.863137 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:32.863098    2566 generic.go:356] "Generic (PLEG): container finished" podID="52b71d8a-53dd-4b07-9cbd-6b08ee2343c3" containerID="76a7b3c469d5bb699aeb4ee3d625b6082479944b4f9276e3c1385b30248a8ecf" exitCode=0
Aug 04 18:10:32.863524 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:32.863174    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s6qrd" event={"ID":"52b71d8a-53dd-4b07-9cbd-6b08ee2343c3","Type":"ContainerDied","Data":"76a7b3c469d5bb699aeb4ee3d625b6082479944b4f9276e3c1385b30248a8ecf"}
Aug 04 18:10:33.869127 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:33.869091    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s6qrd" event={"ID":"52b71d8a-53dd-4b07-9cbd-6b08ee2343c3","Type":"ContainerStarted","Data":"5e4237aafe91864dc314b4c9b41294288ec3404a970fd83a090862286b9a8ccb"}
Aug 04 18:10:33.887970 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:33.887922    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-s6qrd" podStartSLOduration=2.299516347 podStartE2EDuration="3.887905353s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.588389006s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:33.886542703 +0000 UTC m=+633.177156876" watchObservedRunningTime="2026-08-04 18:10:33.887905353 +0000 UTC m=+633.178519523"
Aug 04 18:10:40.901507 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:40.901470    2566 generic.go:356] "Generic (PLEG): container finished" podID="52b71d8a-53dd-4b07-9cbd-6b08ee2343c3" containerID="5e4237aafe91864dc314b4c9b41294288ec3404a970fd83a090862286b9a8ccb" exitCode=0
Aug 04 18:10:40.901975 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:40.901547    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s6qrd" event={"ID":"52b71d8a-53dd-4b07-9cbd-6b08ee2343c3","Type":"ContainerDied","Data":"5e4237aafe91864dc314b4c9b41294288ec3404a970fd83a090862286b9a8ccb"}
Aug 04 18:10:42.059041 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:42.059009    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-s6qrd"
Aug 04 18:10:42.081827 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:42.081793    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52b71d8a-53dd-4b07-9cbd-6b08ee2343c3-model-cache\") pod \"52b71d8a-53dd-4b07-9cbd-6b08ee2343c3\" (UID: \"52b71d8a-53dd-4b07-9cbd-6b08ee2343c3\") "
Aug 04 18:10:42.082006 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:42.081877    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw65x\" (UniqueName: \"kubernetes.io/projected/52b71d8a-53dd-4b07-9cbd-6b08ee2343c3-kube-api-access-tw65x\") pod \"52b71d8a-53dd-4b07-9cbd-6b08ee2343c3\" (UID: \"52b71d8a-53dd-4b07-9cbd-6b08ee2343c3\") "
Aug 04 18:10:42.084218 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:42.084167    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b71d8a-53dd-4b07-9cbd-6b08ee2343c3-kube-api-access-tw65x" (OuterVolumeSpecName: "kube-api-access-tw65x") pod "52b71d8a-53dd-4b07-9cbd-6b08ee2343c3" (UID: "52b71d8a-53dd-4b07-9cbd-6b08ee2343c3"). InnerVolumeSpecName "kube-api-access-tw65x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:10:42.129704 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:42.129664    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b71d8a-53dd-4b07-9cbd-6b08ee2343c3-model-cache" (OuterVolumeSpecName: "model-cache") pod "52b71d8a-53dd-4b07-9cbd-6b08ee2343c3" (UID: "52b71d8a-53dd-4b07-9cbd-6b08ee2343c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:10:42.183264 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:42.183183    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52b71d8a-53dd-4b07-9cbd-6b08ee2343c3-model-cache\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:10:42.183264 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:42.183209    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw65x\" (UniqueName: \"kubernetes.io/projected/52b71d8a-53dd-4b07-9cbd-6b08ee2343c3-kube-api-access-tw65x\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:10:42.911429 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:42.911397    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-s6qrd"
Aug 04 18:10:42.911429 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:42.911404    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s6qrd" event={"ID":"52b71d8a-53dd-4b07-9cbd-6b08ee2343c3","Type":"ContainerDied","Data":"c83adf0454afa51b1346a47e75e42f85f88acda742289d907455d7e73551bb49"}
Aug 04 18:10:42.911429 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:42.911435    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c83adf0454afa51b1346a47e75e42f85f88acda742289d907455d7e73551bb49"
Aug 04 18:10:43.526414 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.526377    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-m65d9"]
Aug 04 18:10:43.526801 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.526703    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52b71d8a-53dd-4b07-9cbd-6b08ee2343c3" containerName="download-hf-model"
Aug 04 18:10:43.526801 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.526713    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="52b71d8a-53dd-4b07-9cbd-6b08ee2343c3" containerName="download-hf-model"
Aug 04 18:10:43.526801 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.526736    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52b71d8a-53dd-4b07-9cbd-6b08ee2343c3" containerName="s3-tls-init-custom"
Aug 04 18:10:43.526801 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.526742    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="52b71d8a-53dd-4b07-9cbd-6b08ee2343c3" containerName="s3-tls-init-custom"
Aug 04 18:10:43.526801 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.526798    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="52b71d8a-53dd-4b07-9cbd-6b08ee2343c3" containerName="s3-tls-init-custom"
Aug 04 18:10:43.528980 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.528965    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-m65d9"
Aug 04 18:10:43.531751 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.531732    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Aug 04 18:10:43.531907 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.531890    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Aug 04 18:10:43.539323 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.539294    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-m65d9"]
Aug 04 18:10:43.594483 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.594449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwpsd\" (UniqueName: \"kubernetes.io/projected/f1fb9e9a-79ff-4cb8-a17d-b1d1699be158-kube-api-access-jwpsd\") pod \"seaweedfs-tls-serving-67b7477c6d-m65d9\" (UID: \"f1fb9e9a-79ff-4cb8-a17d-b1d1699be158\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-m65d9"
Aug 04 18:10:43.594663 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.594512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/f1fb9e9a-79ff-4cb8-a17d-b1d1699be158-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-m65d9\" (UID: \"f1fb9e9a-79ff-4cb8-a17d-b1d1699be158\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-m65d9"
Aug 04 18:10:43.594663 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.594580    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f1fb9e9a-79ff-4cb8-a17d-b1d1699be158-data\") pod \"seaweedfs-tls-serving-67b7477c6d-m65d9\" (UID: \"f1fb9e9a-79ff-4cb8-a17d-b1d1699be158\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-m65d9"
Aug 04 18:10:43.695589 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.695557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/f1fb9e9a-79ff-4cb8-a17d-b1d1699be158-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-m65d9\" (UID: \"f1fb9e9a-79ff-4cb8-a17d-b1d1699be158\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-m65d9"
Aug 04 18:10:43.695761 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.695600    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f1fb9e9a-79ff-4cb8-a17d-b1d1699be158-data\") pod \"seaweedfs-tls-serving-67b7477c6d-m65d9\" (UID: \"f1fb9e9a-79ff-4cb8-a17d-b1d1699be158\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-m65d9"
Aug 04 18:10:43.695761 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.695659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwpsd\" (UniqueName: \"kubernetes.io/projected/f1fb9e9a-79ff-4cb8-a17d-b1d1699be158-kube-api-access-jwpsd\") pod \"seaweedfs-tls-serving-67b7477c6d-m65d9\" (UID: \"f1fb9e9a-79ff-4cb8-a17d-b1d1699be158\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-m65d9"
Aug 04 18:10:43.696111 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.696083    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f1fb9e9a-79ff-4cb8-a17d-b1d1699be158-data\") pod \"seaweedfs-tls-serving-67b7477c6d-m65d9\" (UID: \"f1fb9e9a-79ff-4cb8-a17d-b1d1699be158\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-m65d9"
Aug 04 18:10:43.698108 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.698088    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/f1fb9e9a-79ff-4cb8-a17d-b1d1699be158-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-m65d9\" (UID: \"f1fb9e9a-79ff-4cb8-a17d-b1d1699be158\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-m65d9"
Aug 04 18:10:43.706548 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.706522    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwpsd\" (UniqueName: \"kubernetes.io/projected/f1fb9e9a-79ff-4cb8-a17d-b1d1699be158-kube-api-access-jwpsd\") pod \"seaweedfs-tls-serving-67b7477c6d-m65d9\" (UID: \"f1fb9e9a-79ff-4cb8-a17d-b1d1699be158\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-m65d9"
Aug 04 18:10:43.838996 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.838967    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-m65d9"
Aug 04 18:10:43.975800 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:43.975771    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-m65d9"]
Aug 04 18:10:43.978348 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:10:43.978318    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fb9e9a_79ff_4cb8_a17d_b1d1699be158.slice/crio-9785f7568eaf4d428aba1277f22f78fdbe1a16c7c3aea68dce31d0622b277614 WatchSource:0}: Error finding container 9785f7568eaf4d428aba1277f22f78fdbe1a16c7c3aea68dce31d0622b277614: Status 404 returned error can't find the container with id 9785f7568eaf4d428aba1277f22f78fdbe1a16c7c3aea68dce31d0622b277614
Aug 04 18:10:44.921721 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:44.921686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-m65d9" event={"ID":"f1fb9e9a-79ff-4cb8-a17d-b1d1699be158","Type":"ContainerStarted","Data":"868db6d00c87310a1b1a40b00b5bf9b93e0479a466ee6ba98e71807549880e21"}
Aug 04 18:10:44.922099 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:44.921737    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-m65d9" event={"ID":"f1fb9e9a-79ff-4cb8-a17d-b1d1699be158","Type":"ContainerStarted","Data":"9785f7568eaf4d428aba1277f22f78fdbe1a16c7c3aea68dce31d0622b277614"}
Aug 04 18:10:44.944873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:44.944823    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-m65d9" podStartSLOduration=1.6760970579999999 podStartE2EDuration="1.944807254s" totalImagesPullingTime="268.710196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:44.943776674 +0000 UTC m=+644.234390846" watchObservedRunningTime="2026-08-04 18:10:44.944807254 +0000 UTC m=+644.235421464"
Aug 04 18:10:45.522412 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:45.522377    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-v6pwv"]
Aug 04 18:10:45.525326 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:45.525308    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-v6pwv"
Aug 04 18:10:45.534556 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:45.534530    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-v6pwv"]
Aug 04 18:10:45.611748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:45.611717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlkw8\" (UniqueName: \"kubernetes.io/projected/f4a5204e-1056-4c95-8c20-14ecde19ac08-kube-api-access-qlkw8\") pod \"s3-tls-init-serving-v6pwv\" (UID: \"f4a5204e-1056-4c95-8c20-14ecde19ac08\") " pod="kserve/s3-tls-init-serving-v6pwv"
Aug 04 18:10:45.611748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:45.611753    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4a5204e-1056-4c95-8c20-14ecde19ac08-model-cache\") pod \"s3-tls-init-serving-v6pwv\" (UID: \"f4a5204e-1056-4c95-8c20-14ecde19ac08\") " pod="kserve/s3-tls-init-serving-v6pwv"
Aug 04 18:10:45.712320 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:45.712284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlkw8\" (UniqueName: \"kubernetes.io/projected/f4a5204e-1056-4c95-8c20-14ecde19ac08-kube-api-access-qlkw8\") pod \"s3-tls-init-serving-v6pwv\" (UID: \"f4a5204e-1056-4c95-8c20-14ecde19ac08\") " pod="kserve/s3-tls-init-serving-v6pwv"
Aug 04 18:10:45.712320 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:45.712322    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4a5204e-1056-4c95-8c20-14ecde19ac08-model-cache\") pod \"s3-tls-init-serving-v6pwv\" (UID: \"f4a5204e-1056-4c95-8c20-14ecde19ac08\") " pod="kserve/s3-tls-init-serving-v6pwv"
Aug 04 18:10:45.712679 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:45.712663    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4a5204e-1056-4c95-8c20-14ecde19ac08-model-cache\") pod \"s3-tls-init-serving-v6pwv\" (UID: \"f4a5204e-1056-4c95-8c20-14ecde19ac08\") " pod="kserve/s3-tls-init-serving-v6pwv"
Aug 04 18:10:45.723424 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:45.723399    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlkw8\" (UniqueName: \"kubernetes.io/projected/f4a5204e-1056-4c95-8c20-14ecde19ac08-kube-api-access-qlkw8\") pod \"s3-tls-init-serving-v6pwv\" (UID: \"f4a5204e-1056-4c95-8c20-14ecde19ac08\") " pod="kserve/s3-tls-init-serving-v6pwv"
Aug 04 18:10:45.841130 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:45.841083    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-v6pwv"
Aug 04 18:10:45.975632 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:45.975608    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-v6pwv"]
Aug 04 18:10:45.977211 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:10:45.977182    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4a5204e_1056_4c95_8c20_14ecde19ac08.slice/crio-79cf8ad3732b2bdc63038c9e9a92989a0575f0aa2f5d4e735241f0239397fd7b WatchSource:0}: Error finding container 79cf8ad3732b2bdc63038c9e9a92989a0575f0aa2f5d4e735241f0239397fd7b: Status 404 returned error can't find the container with id 79cf8ad3732b2bdc63038c9e9a92989a0575f0aa2f5d4e735241f0239397fd7b
Aug 04 18:10:46.933945 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:46.933903    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-v6pwv" event={"ID":"f4a5204e-1056-4c95-8c20-14ecde19ac08","Type":"ContainerStarted","Data":"9fd08f8cfd3708bea2a6933e1ec45971fb04b2f57df0f878f65e35f2d9014edd"}
Aug 04 18:10:46.934204 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:46.933959    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-v6pwv" event={"ID":"f4a5204e-1056-4c95-8c20-14ecde19ac08","Type":"ContainerStarted","Data":"79cf8ad3732b2bdc63038c9e9a92989a0575f0aa2f5d4e735241f0239397fd7b"}
Aug 04 18:10:47.939402 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:47.939346    2566 generic.go:356] "Generic (PLEG): container finished" podID="f4a5204e-1056-4c95-8c20-14ecde19ac08" containerID="9fd08f8cfd3708bea2a6933e1ec45971fb04b2f57df0f878f65e35f2d9014edd" exitCode=0
Aug 04 18:10:47.939895 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:47.939453    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-v6pwv" event={"ID":"f4a5204e-1056-4c95-8c20-14ecde19ac08","Type":"ContainerDied","Data":"9fd08f8cfd3708bea2a6933e1ec45971fb04b2f57df0f878f65e35f2d9014edd"}
Aug 04 18:10:48.945135 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:48.945097    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-v6pwv" event={"ID":"f4a5204e-1056-4c95-8c20-14ecde19ac08","Type":"ContainerStarted","Data":"0cb1ca9657268e9f5a868c235cabf72a19065489629eacc3763f0a9b6a10623c"}
Aug 04 18:10:48.969369 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:48.969267    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-v6pwv" podStartSLOduration=2.465323822 podStartE2EDuration="3.969245986s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.503922164s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:48.968839548 +0000 UTC m=+648.259453722" watchObservedRunningTime="2026-08-04 18:10:48.969245986 +0000 UTC m=+648.259860157"
Aug 04 18:10:53.977891 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:53.977856    2566 generic.go:356] "Generic (PLEG): container finished" podID="f4a5204e-1056-4c95-8c20-14ecde19ac08" containerID="0cb1ca9657268e9f5a868c235cabf72a19065489629eacc3763f0a9b6a10623c" exitCode=0
Aug 04 18:10:53.978265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:53.977931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-v6pwv" event={"ID":"f4a5204e-1056-4c95-8c20-14ecde19ac08","Type":"ContainerDied","Data":"0cb1ca9657268e9f5a868c235cabf72a19065489629eacc3763f0a9b6a10623c"}
Aug 04 18:10:55.149605 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:55.149584    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-v6pwv"
Aug 04 18:10:55.200742 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:55.200716    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlkw8\" (UniqueName: \"kubernetes.io/projected/f4a5204e-1056-4c95-8c20-14ecde19ac08-kube-api-access-qlkw8\") pod \"f4a5204e-1056-4c95-8c20-14ecde19ac08\" (UID: \"f4a5204e-1056-4c95-8c20-14ecde19ac08\") "
Aug 04 18:10:55.200912 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:55.200778    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4a5204e-1056-4c95-8c20-14ecde19ac08-model-cache\") pod \"f4a5204e-1056-4c95-8c20-14ecde19ac08\" (UID: \"f4a5204e-1056-4c95-8c20-14ecde19ac08\") "
Aug 04 18:10:55.202696 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:55.202657    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a5204e-1056-4c95-8c20-14ecde19ac08-kube-api-access-qlkw8" (OuterVolumeSpecName: "kube-api-access-qlkw8") pod "f4a5204e-1056-4c95-8c20-14ecde19ac08" (UID: "f4a5204e-1056-4c95-8c20-14ecde19ac08"). InnerVolumeSpecName "kube-api-access-qlkw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:10:55.223014 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:55.222979    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a5204e-1056-4c95-8c20-14ecde19ac08-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4a5204e-1056-4c95-8c20-14ecde19ac08" (UID: "f4a5204e-1056-4c95-8c20-14ecde19ac08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:10:55.301967 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:55.301939    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlkw8\" (UniqueName: \"kubernetes.io/projected/f4a5204e-1056-4c95-8c20-14ecde19ac08-kube-api-access-qlkw8\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:10:55.302094 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:55.301969    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4a5204e-1056-4c95-8c20-14ecde19ac08-model-cache\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:10:55.988807 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:55.988782    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-v6pwv"
Aug 04 18:10:55.988995 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:55.988813    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-v6pwv" event={"ID":"f4a5204e-1056-4c95-8c20-14ecde19ac08","Type":"ContainerDied","Data":"79cf8ad3732b2bdc63038c9e9a92989a0575f0aa2f5d4e735241f0239397fd7b"}
Aug 04 18:10:55.988995 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:10:55.988845    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79cf8ad3732b2bdc63038c9e9a92989a0575f0aa2f5d4e735241f0239397fd7b"
Aug 04 18:11:09.016756 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.016719    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"]
Aug 04 18:11:09.017131 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.017076    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a5204e-1056-4c95-8c20-14ecde19ac08" containerName="s3-tls-init-serving"
Aug 04 18:11:09.017131 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.017086    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a5204e-1056-4c95-8c20-14ecde19ac08" containerName="s3-tls-init-serving"
Aug 04 18:11:09.017131 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.017100    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a5204e-1056-4c95-8c20-14ecde19ac08" containerName="download-hf-model"
Aug 04 18:11:09.017131 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.017107    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a5204e-1056-4c95-8c20-14ecde19ac08" containerName="download-hf-model"
Aug 04 18:11:09.017259 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.017164    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a5204e-1056-4c95-8c20-14ecde19ac08" containerName="s3-tls-init-serving"
Aug 04 18:11:09.020187 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.020172    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.022798 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.022775    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Aug 04 18:11:09.022929 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.022819    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:11:09.022929 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.022915    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Aug 04 18:11:09.023114 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.023093    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:11:09.023176 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.023163    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:11:09.024328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.024305    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:11:09.024449 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.024337    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:11:09.034032 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.034009    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"]
Aug 04 18:11:09.129834 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.129795    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bec084e-18cb-46f7-8479-e6081c14f0b9-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.129993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.129850    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bec084e-18cb-46f7-8479-e6081c14f0b9-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.129993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.129873    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bec084e-18cb-46f7-8479-e6081c14f0b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.129993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.129942    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bec084e-18cb-46f7-8479-e6081c14f0b9-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.129993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.129975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw5l2\" (UniqueName: \"kubernetes.io/projected/7bec084e-18cb-46f7-8479-e6081c14f0b9-kube-api-access-tw5l2\") pod \"isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.175274 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.175241    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"]
Aug 04 18:11:09.177973 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.177954    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.180534 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.180514    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Aug 04 18:11:09.180658 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.180514    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Aug 04 18:11:09.193875 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.193849    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"]
Aug 04 18:11:09.231085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.231051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bec084e-18cb-46f7-8479-e6081c14f0b9-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.231255 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.231105    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bec084e-18cb-46f7-8479-e6081c14f0b9-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.231255 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.231134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bec084e-18cb-46f7-8479-e6081c14f0b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.231255 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.231169    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bec084e-18cb-46f7-8479-e6081c14f0b9-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.231255 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.231195    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw5l2\" (UniqueName: \"kubernetes.io/projected/7bec084e-18cb-46f7-8479-e6081c14f0b9-kube-api-access-tw5l2\") pod \"isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.231449 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:11:09.231306    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Aug 04 18:11:09.231449 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:11:09.231436    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bec084e-18cb-46f7-8479-e6081c14f0b9-proxy-tls podName:7bec084e-18cb-46f7-8479-e6081c14f0b9 nodeName:}" failed. No retries permitted until 2026-08-04 18:11:09.731413343 +0000 UTC m=+669.022027495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7bec084e-18cb-46f7-8479-e6081c14f0b9-proxy-tls") pod "isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" (UID: "7bec084e-18cb-46f7-8479-e6081c14f0b9") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Aug 04 18:11:09.231581 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.231525    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bec084e-18cb-46f7-8479-e6081c14f0b9-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.231872 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.231856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bec084e-18cb-46f7-8479-e6081c14f0b9-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.231910 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.231880    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bec084e-18cb-46f7-8479-e6081c14f0b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.245175 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.245148    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw5l2\" (UniqueName: \"kubernetes.io/projected/7bec084e-18cb-46f7-8479-e6081c14f0b9-kube-api-access-tw5l2\") pod \"isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.331760 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.331695    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.331760 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.331764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.331974 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.331848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvjxn\" (UniqueName: \"kubernetes.io/projected/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kube-api-access-pvjxn\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.331974 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.331940    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.332041 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.331998    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.432808 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.432776    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.432808 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.432808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.433009 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.432846    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjxn\" (UniqueName: \"kubernetes.io/projected/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kube-api-access-pvjxn\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.433009 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.432910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.433009 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.432991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.433331 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.433307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.433604 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.433585    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.433649 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.433583    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.435282 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.435265    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.444000 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.443977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjxn\" (UniqueName: \"kubernetes.io/projected/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kube-api-access-pvjxn\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.490225 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.490182    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:09.635852 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.633720    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"]
Aug 04 18:11:09.640596 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:11:09.640549    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f530f5b_5bc2_4f29_a296_2d6b055a30e5.slice/crio-3413fce54a5710ae470272b12260c92273935689640b971460b80fdaf52694a3 WatchSource:0}: Error finding container 3413fce54a5710ae470272b12260c92273935689640b971460b80fdaf52694a3: Status 404 returned error can't find the container with id 3413fce54a5710ae470272b12260c92273935689640b971460b80fdaf52694a3
Aug 04 18:11:09.738581 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.738546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bec084e-18cb-46f7-8479-e6081c14f0b9-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.740894 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.740863    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bec084e-18cb-46f7-8479-e6081c14f0b9-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:09.930736 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:09.930655    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:10.052745 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:10.052707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" event={"ID":"4f530f5b-5bc2-4f29-a296-2d6b055a30e5","Type":"ContainerStarted","Data":"fcdbf41134c6c76524f0cb90ac6a14f9310b310d71e256bd25770f0ee8fcfb78"}
Aug 04 18:11:10.053163 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:10.052788    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" event={"ID":"4f530f5b-5bc2-4f29-a296-2d6b055a30e5","Type":"ContainerStarted","Data":"3413fce54a5710ae470272b12260c92273935689640b971460b80fdaf52694a3"}
Aug 04 18:11:10.072765 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:10.072739    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"]
Aug 04 18:11:10.074305 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:11:10.074285    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bec084e_18cb_46f7_8479_e6081c14f0b9.slice/crio-b864768c871ff72f79fb01c795f4c1281249c25d3f3fe06ef04d52f409f3f86a WatchSource:0}: Error finding container b864768c871ff72f79fb01c795f4c1281249c25d3f3fe06ef04d52f409f3f86a: Status 404 returned error can't find the container with id b864768c871ff72f79fb01c795f4c1281249c25d3f3fe06ef04d52f409f3f86a
Aug 04 18:11:11.059911 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:11.059866    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" event={"ID":"7bec084e-18cb-46f7-8479-e6081c14f0b9","Type":"ContainerStarted","Data":"b6ed138164ef8826c2f2e75a6662001ec66c4c560021dabc502eb91051004f0d"}
Aug 04 18:11:11.060410 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:11.059952    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" event={"ID":"7bec084e-18cb-46f7-8479-e6081c14f0b9","Type":"ContainerStarted","Data":"b864768c871ff72f79fb01c795f4c1281249c25d3f3fe06ef04d52f409f3f86a"}
Aug 04 18:11:14.076129 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:14.076094    2566 generic.go:356] "Generic (PLEG): container finished" podID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerID="b6ed138164ef8826c2f2e75a6662001ec66c4c560021dabc502eb91051004f0d" exitCode=0
Aug 04 18:11:14.076569 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:14.076165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" event={"ID":"7bec084e-18cb-46f7-8479-e6081c14f0b9","Type":"ContainerDied","Data":"b6ed138164ef8826c2f2e75a6662001ec66c4c560021dabc502eb91051004f0d"}
Aug 04 18:11:14.077765 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:14.077743    2566 generic.go:356] "Generic (PLEG): container finished" podID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerID="fcdbf41134c6c76524f0cb90ac6a14f9310b310d71e256bd25770f0ee8fcfb78" exitCode=0
Aug 04 18:11:14.077880 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:14.077853    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" event={"ID":"4f530f5b-5bc2-4f29-a296-2d6b055a30e5","Type":"ContainerDied","Data":"fcdbf41134c6c76524f0cb90ac6a14f9310b310d71e256bd25770f0ee8fcfb78"}
Aug 04 18:11:34.238744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:34.238713    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" event={"ID":"4f530f5b-5bc2-4f29-a296-2d6b055a30e5","Type":"ContainerStarted","Data":"e7fbc328137c7467cbd4c450ba8870519cfd8259a814fd9d308c533838c6c1bc"}
Aug 04 18:11:34.240488 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:34.240461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" event={"ID":"7bec084e-18cb-46f7-8479-e6081c14f0b9","Type":"ContainerStarted","Data":"37be873247c37ba348c02e849fe347cbe38331bd5d73baa6634fc0f91d8b0172"}
Aug 04 18:11:37.263297 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:37.263263    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" event={"ID":"7bec084e-18cb-46f7-8479-e6081c14f0b9","Type":"ContainerStarted","Data":"fbe3a111ebb34458c4fe914b797e19b70be03eda617bcbbcf50659c79489f89b"}
Aug 04 18:11:37.265759 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:37.265731    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" event={"ID":"4f530f5b-5bc2-4f29-a296-2d6b055a30e5","Type":"ContainerStarted","Data":"47e17cdedd5c4fdb11579475df55c7691bae20726a9339e3c9fa798c292392d6"}
Aug 04 18:11:37.265913 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:37.265894    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:37.292160 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:37.292105    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" podStartSLOduration=2.028915457 podStartE2EDuration="28.292088664s" totalImagesPullingTime="22.484720297s" totalInitContainerRuntime="3.77845291s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:37.290388538 +0000 UTC m=+696.581002733" watchObservedRunningTime="2026-08-04 18:11:37.292088664 +0000 UTC m=+696.582702834"
Aug 04 18:11:38.271575 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:38.271533    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:38.272800 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:38.272754    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 04 18:11:39.277254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:39.277207    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 04 18:11:40.283477 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:40.283444    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" event={"ID":"7bec084e-18cb-46f7-8479-e6081c14f0b9","Type":"ContainerStarted","Data":"ed686af051dfa5566a294aac62a61c471342f0f49c5fb62408c87e4ffcb618a5"}
Aug 04 18:11:40.283884 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:40.283563    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:40.320399 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:40.320311    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podStartSLOduration=3.305738034 podStartE2EDuration="32.320294791s" totalImagesPullingTime="25.365190509s" totalInitContainerRuntime="3.649366248s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:40.320201792 +0000 UTC m=+699.610815973" watchObservedRunningTime="2026-08-04 18:11:40.320294791 +0000 UTC m=+699.610908961"
Aug 04 18:11:41.288471 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:41.288429    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:41.288965 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:41.288511    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:41.290184 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:41.290146    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 04 18:11:41.290926 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:41.290886    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:11:41.293753 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:41.293733    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:11:42.292908 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:42.292852    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 04 18:11:42.293328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:42.293306    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:11:43.297006 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:43.296966    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 04 18:11:43.297445 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:43.297263    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:11:44.282411 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:44.282384    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:11:44.282893 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:44.282865    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 04 18:11:53.296969 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:53.296875    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 04 18:11:53.297425 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:53.297272    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:11:54.283658 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:11:54.283618    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 04 18:12:03.297208 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:03.297150    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 04 18:12:03.297687 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:03.297670    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:12:04.283472 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:04.283431    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 04 18:12:13.297782 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:13.297740    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 04 18:12:13.298194 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:13.298176    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:12:14.283656 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:14.283615    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 04 18:12:23.297094 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:23.297041    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 04 18:12:23.297630 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:23.297607    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:12:24.283615 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:24.283576    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 04 18:12:33.297639 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:33.297589    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 04 18:12:33.298122 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:33.298049    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:12:34.283373 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:34.283323    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 04 18:12:43.297850 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:43.297803    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 04 18:12:43.298285 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:43.298259    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:12:44.282992 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:44.282955    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 04 18:12:53.298038 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:53.298011    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:12:53.298455 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:53.298147    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:12:54.283505 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:54.283469    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 04 18:12:55.194146 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:12:55.194104    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 04 18:13:04.141276 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.141242    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"]
Aug 04 18:13:04.141707 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.141567    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kserve-container" containerID="cri-o://37be873247c37ba348c02e849fe347cbe38331bd5d73baa6634fc0f91d8b0172" gracePeriod=30
Aug 04 18:13:04.141707 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.141630    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kube-rbac-proxy" containerID="cri-o://fbe3a111ebb34458c4fe914b797e19b70be03eda617bcbbcf50659c79489f89b" gracePeriod=30
Aug 04 18:13:04.141906 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.141574    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="agent" containerID="cri-o://ed686af051dfa5566a294aac62a61c471342f0f49c5fb62408c87e4ffcb618a5" gracePeriod=30
Aug 04 18:13:04.217554 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.217519    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"]
Aug 04 18:13:04.220298 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.220282    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:04.222525 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.222501    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Aug 04 18:13:04.222525 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.222520    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Aug 04 18:13:04.236091 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.236060    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"]
Aug 04 18:13:04.347244 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.347212    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btvcm\" (UniqueName: \"kubernetes.io/projected/452be1b6-5f5b-479d-a422-45532c22117d-kube-api-access-btvcm\") pod \"isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:04.347244 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.347249    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/452be1b6-5f5b-479d-a422-45532c22117d-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:04.347556 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.347272    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/452be1b6-5f5b-479d-a422-45532c22117d-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:04.347556 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.347393    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/452be1b6-5f5b-479d-a422-45532c22117d-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:04.347556 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.347467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/452be1b6-5f5b-479d-a422-45532c22117d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:04.448154 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.448048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btvcm\" (UniqueName: \"kubernetes.io/projected/452be1b6-5f5b-479d-a422-45532c22117d-kube-api-access-btvcm\") pod \"isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:04.448154 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.448108    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/452be1b6-5f5b-479d-a422-45532c22117d-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:04.448154 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.448144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/452be1b6-5f5b-479d-a422-45532c22117d-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:04.448482 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.448205    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/452be1b6-5f5b-479d-a422-45532c22117d-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:04.448482 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.448284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/452be1b6-5f5b-479d-a422-45532c22117d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:04.448482 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:13:04.448291    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Aug 04 18:13:04.448482 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:13:04.448406    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/452be1b6-5f5b-479d-a422-45532c22117d-proxy-tls podName:452be1b6-5f5b-479d-a422-45532c22117d nodeName:}" failed. No retries permitted until 2026-08-04 18:13:04.948388346 +0000 UTC m=+784.239002495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/452be1b6-5f5b-479d-a422-45532c22117d-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" (UID: "452be1b6-5f5b-479d-a422-45532c22117d") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Aug 04 18:13:04.448725 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.448705    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/452be1b6-5f5b-479d-a422-45532c22117d-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:04.448848 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.448830    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/452be1b6-5f5b-479d-a422-45532c22117d-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:04.448971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.448952    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/452be1b6-5f5b-479d-a422-45532c22117d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:04.461335 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.461310    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btvcm\" (UniqueName: \"kubernetes.io/projected/452be1b6-5f5b-479d-a422-45532c22117d-kube-api-access-btvcm\") pod \"isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:04.699461 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.699385    2566 generic.go:356] "Generic (PLEG): container finished" podID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerID="fbe3a111ebb34458c4fe914b797e19b70be03eda617bcbbcf50659c79489f89b" exitCode=2
Aug 04 18:13:04.699461 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.699446    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" event={"ID":"7bec084e-18cb-46f7-8479-e6081c14f0b9","Type":"ContainerDied","Data":"fbe3a111ebb34458c4fe914b797e19b70be03eda617bcbbcf50659c79489f89b"}
Aug 04 18:13:04.953718 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.953630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/452be1b6-5f5b-479d-a422-45532c22117d-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:04.956081 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:04.956052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/452be1b6-5f5b-479d-a422-45532c22117d-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:05.131327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:05.131277    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:05.198795 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:05.198770    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:13:05.269232 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:05.269204    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"]
Aug 04 18:13:05.270319 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:13:05.270290    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452be1b6_5f5b_479d_a422_45532c22117d.slice/crio-20d9403367969e88d7410a3788bd80600e5f66574a3cd8a11de4f917efc3590c WatchSource:0}: Error finding container 20d9403367969e88d7410a3788bd80600e5f66574a3cd8a11de4f917efc3590c: Status 404 returned error can't find the container with id 20d9403367969e88d7410a3788bd80600e5f66574a3cd8a11de4f917efc3590c
Aug 04 18:13:05.706655 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:05.706620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" event={"ID":"452be1b6-5f5b-479d-a422-45532c22117d","Type":"ContainerStarted","Data":"bad314ea4729d775977633086926d41c3bbccb8778e6c7a2ab8715a7d183e00d"}
Aug 04 18:13:05.706946 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:05.706702    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" event={"ID":"452be1b6-5f5b-479d-a422-45532c22117d","Type":"ContainerStarted","Data":"20d9403367969e88d7410a3788bd80600e5f66574a3cd8a11de4f917efc3590c"}
Aug 04 18:13:06.289036 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:06.288995    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Aug 04 18:13:09.727958 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:09.727923    2566 generic.go:356] "Generic (PLEG): container finished" podID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerID="37be873247c37ba348c02e849fe347cbe38331bd5d73baa6634fc0f91d8b0172" exitCode=0
Aug 04 18:13:09.728415 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:09.727997    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" event={"ID":"7bec084e-18cb-46f7-8479-e6081c14f0b9","Type":"ContainerDied","Data":"37be873247c37ba348c02e849fe347cbe38331bd5d73baa6634fc0f91d8b0172"}
Aug 04 18:13:09.729473 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:09.729451    2566 generic.go:356] "Generic (PLEG): container finished" podID="452be1b6-5f5b-479d-a422-45532c22117d" containerID="bad314ea4729d775977633086926d41c3bbccb8778e6c7a2ab8715a7d183e00d" exitCode=0
Aug 04 18:13:09.729578 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:09.729506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" event={"ID":"452be1b6-5f5b-479d-a422-45532c22117d","Type":"ContainerDied","Data":"bad314ea4729d775977633086926d41c3bbccb8778e6c7a2ab8715a7d183e00d"}
Aug 04 18:13:10.736985 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:10.736946    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" event={"ID":"452be1b6-5f5b-479d-a422-45532c22117d","Type":"ContainerStarted","Data":"7649a2364a43cda604be4d413e999fdeb5574cb5906d3ca7be12fc4b667d4cc1"}
Aug 04 18:13:10.737403 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:10.737057    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" event={"ID":"452be1b6-5f5b-479d-a422-45532c22117d","Type":"ContainerStarted","Data":"10a69171e313da94bc522cee19e008688ce9e49b2e7e4494678ec6d7c1b23f98"}
Aug 04 18:13:10.737403 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:10.737084    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" event={"ID":"452be1b6-5f5b-479d-a422-45532c22117d","Type":"ContainerStarted","Data":"2219777eb7f93da198db09aac3f57e8c81174f3dc10ccc748db9af75f753b3b9"}
Aug 04 18:13:10.737403 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:10.737380    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:10.737573 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:10.737526    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:10.739020 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:10.738993    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Aug 04 18:13:10.767411 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:10.767342    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podStartSLOduration=3.223056983 podStartE2EDuration="6.767323564s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.544266581s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:13:10.76518797 +0000 UTC m=+790.055802141" watchObservedRunningTime="2026-08-04 18:13:10.767323564 +0000 UTC m=+790.057937734"
Aug 04 18:13:11.288867 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:11.288820    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Aug 04 18:13:11.742139 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:11.742101    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Aug 04 18:13:11.742547 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:11.742115    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:11.743283 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:11.743256    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:13:12.746961 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:12.746919    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Aug 04 18:13:12.747450 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:12.747430    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:13:13.296993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:13.296946    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 04 18:13:13.297271 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:13.297244    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:13:14.075856 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:14.075822    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"]
Aug 04 18:13:14.076275 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:14.076092    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kserve-container" containerID="cri-o://e7fbc328137c7467cbd4c450ba8870519cfd8259a814fd9d308c533838c6c1bc" gracePeriod=30
Aug 04 18:13:14.076275 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:14.076144    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kube-rbac-proxy" containerID="cri-o://47e17cdedd5c4fdb11579475df55c7691bae20726a9339e3c9fa798c292392d6" gracePeriod=30
Aug 04 18:13:14.278499 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:14.278459    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Aug 04 18:13:14.759571 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:14.759538    2566 generic.go:356] "Generic (PLEG): container finished" podID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerID="47e17cdedd5c4fdb11579475df55c7691bae20726a9339e3c9fa798c292392d6" exitCode=2
Aug 04 18:13:14.759740 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:14.759590    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" event={"ID":"4f530f5b-5bc2-4f29-a296-2d6b055a30e5","Type":"ContainerDied","Data":"47e17cdedd5c4fdb11579475df55c7691bae20726a9339e3c9fa798c292392d6"}
Aug 04 18:13:15.194173 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:15.194137    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Aug 04 18:13:16.289750 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:16.289707    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Aug 04 18:13:16.290220 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:16.289837    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:13:17.752401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:17.752376    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:13:17.753191 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:17.753124    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Aug 04 18:13:17.753633 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:17.753607    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:13:17.920651 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:17.920627    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:13:17.975637 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:17.975611    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kserve-provision-location\") pod \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") "
Aug 04 18:13:17.975794 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:17.975662    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kube-rbac-proxy-sar-config\") pod \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") "
Aug 04 18:13:17.975794 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:17.975715    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-cabundle-cert\") pod \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") "
Aug 04 18:13:17.975794 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:17.975750    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-proxy-tls\") pod \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") "
Aug 04 18:13:17.975794 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:17.975780    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvjxn\" (UniqueName: \"kubernetes.io/projected/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kube-api-access-pvjxn\") pod \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\" (UID: \"4f530f5b-5bc2-4f29-a296-2d6b055a30e5\") "
Aug 04 18:13:17.976028 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:17.975980    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f530f5b-5bc2-4f29-a296-2d6b055a30e5" (UID: "4f530f5b-5bc2-4f29-a296-2d6b055a30e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:13:17.976092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:17.976027    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f530f5b-5bc2-4f29-a296-2d6b055a30e5" (UID: "4f530f5b-5bc2-4f29-a296-2d6b055a30e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:13:17.976137 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:17.976085    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f530f5b-5bc2-4f29-a296-2d6b055a30e5" (UID: "4f530f5b-5bc2-4f29-a296-2d6b055a30e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:13:17.977853 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:17.977828    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f530f5b-5bc2-4f29-a296-2d6b055a30e5" (UID: "4f530f5b-5bc2-4f29-a296-2d6b055a30e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:13:17.978114 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:17.977885    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kube-api-access-pvjxn" (OuterVolumeSpecName: "kube-api-access-pvjxn") pod "4f530f5b-5bc2-4f29-a296-2d6b055a30e5" (UID: "4f530f5b-5bc2-4f29-a296-2d6b055a30e5"). InnerVolumeSpecName "kube-api-access-pvjxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:13:18.077443 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.077410    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:13:18.077443 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.077440    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:13:18.077662 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.077453    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:13:18.077662 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.077465    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:13:18.077662 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.077478    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvjxn\" (UniqueName: \"kubernetes.io/projected/4f530f5b-5bc2-4f29-a296-2d6b055a30e5-kube-api-access-pvjxn\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:13:18.781370 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.781317    2566 generic.go:356] "Generic (PLEG): container finished" podID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerID="e7fbc328137c7467cbd4c450ba8870519cfd8259a814fd9d308c533838c6c1bc" exitCode=0
Aug 04 18:13:18.781806 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.781384    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"
Aug 04 18:13:18.781806 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.781386    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" event={"ID":"4f530f5b-5bc2-4f29-a296-2d6b055a30e5","Type":"ContainerDied","Data":"e7fbc328137c7467cbd4c450ba8870519cfd8259a814fd9d308c533838c6c1bc"}
Aug 04 18:13:18.781806 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.781454    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z" event={"ID":"4f530f5b-5bc2-4f29-a296-2d6b055a30e5","Type":"ContainerDied","Data":"3413fce54a5710ae470272b12260c92273935689640b971460b80fdaf52694a3"}
Aug 04 18:13:18.781806 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.781483    2566 scope.go:117] "RemoveContainer" containerID="47e17cdedd5c4fdb11579475df55c7691bae20726a9339e3c9fa798c292392d6"
Aug 04 18:13:18.790617 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.790602    2566 scope.go:117] "RemoveContainer" containerID="e7fbc328137c7467cbd4c450ba8870519cfd8259a814fd9d308c533838c6c1bc"
Aug 04 18:13:18.798599 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.798380    2566 scope.go:117] "RemoveContainer" containerID="fcdbf41134c6c76524f0cb90ac6a14f9310b310d71e256bd25770f0ee8fcfb78"
Aug 04 18:13:18.804713 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.804687    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"]
Aug 04 18:13:18.806543 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.806528    2566 scope.go:117] "RemoveContainer" containerID="47e17cdedd5c4fdb11579475df55c7691bae20726a9339e3c9fa798c292392d6"
Aug 04 18:13:18.806850 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:13:18.806831    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e17cdedd5c4fdb11579475df55c7691bae20726a9339e3c9fa798c292392d6\": container with ID starting with 47e17cdedd5c4fdb11579475df55c7691bae20726a9339e3c9fa798c292392d6 not found: ID does not exist" containerID="47e17cdedd5c4fdb11579475df55c7691bae20726a9339e3c9fa798c292392d6"
Aug 04 18:13:18.806900 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.806860    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e17cdedd5c4fdb11579475df55c7691bae20726a9339e3c9fa798c292392d6"} err="failed to get container status \"47e17cdedd5c4fdb11579475df55c7691bae20726a9339e3c9fa798c292392d6\": rpc error: code = NotFound desc = could not find container \"47e17cdedd5c4fdb11579475df55c7691bae20726a9339e3c9fa798c292392d6\": container with ID starting with 47e17cdedd5c4fdb11579475df55c7691bae20726a9339e3c9fa798c292392d6 not found: ID does not exist"
Aug 04 18:13:18.806900 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.806876    2566 scope.go:117] "RemoveContainer" containerID="e7fbc328137c7467cbd4c450ba8870519cfd8259a814fd9d308c533838c6c1bc"
Aug 04 18:13:18.807116 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:13:18.807099    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7fbc328137c7467cbd4c450ba8870519cfd8259a814fd9d308c533838c6c1bc\": container with ID starting with e7fbc328137c7467cbd4c450ba8870519cfd8259a814fd9d308c533838c6c1bc not found: ID does not exist" containerID="e7fbc328137c7467cbd4c450ba8870519cfd8259a814fd9d308c533838c6c1bc"
Aug 04 18:13:18.807154 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.807124    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7fbc328137c7467cbd4c450ba8870519cfd8259a814fd9d308c533838c6c1bc"} err="failed to get container status \"e7fbc328137c7467cbd4c450ba8870519cfd8259a814fd9d308c533838c6c1bc\": rpc error: code = NotFound desc = could not find container \"e7fbc328137c7467cbd4c450ba8870519cfd8259a814fd9d308c533838c6c1bc\": container with ID starting with e7fbc328137c7467cbd4c450ba8870519cfd8259a814fd9d308c533838c6c1bc not found: ID does not exist"
Aug 04 18:13:18.807154 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.807141    2566 scope.go:117] "RemoveContainer" containerID="fcdbf41134c6c76524f0cb90ac6a14f9310b310d71e256bd25770f0ee8fcfb78"
Aug 04 18:13:18.807421 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:13:18.807400    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcdbf41134c6c76524f0cb90ac6a14f9310b310d71e256bd25770f0ee8fcfb78\": container with ID starting with fcdbf41134c6c76524f0cb90ac6a14f9310b310d71e256bd25770f0ee8fcfb78 not found: ID does not exist" containerID="fcdbf41134c6c76524f0cb90ac6a14f9310b310d71e256bd25770f0ee8fcfb78"
Aug 04 18:13:18.807470 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.807430    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcdbf41134c6c76524f0cb90ac6a14f9310b310d71e256bd25770f0ee8fcfb78"} err="failed to get container status \"fcdbf41134c6c76524f0cb90ac6a14f9310b310d71e256bd25770f0ee8fcfb78\": rpc error: code = NotFound desc = could not find container \"fcdbf41134c6c76524f0cb90ac6a14f9310b310d71e256bd25770f0ee8fcfb78\": container with ID starting with fcdbf41134c6c76524f0cb90ac6a14f9310b310d71e256bd25770f0ee8fcfb78 not found: ID does not exist"
Aug 04 18:13:18.812944 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:18.812923    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-vqh2z"]
Aug 04 18:13:19.195467 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.195438    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" path="/var/lib/kubelet/pods/4f530f5b-5bc2-4f29-a296-2d6b055a30e5/volumes"
Aug 04 18:13:19.195779 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.195766    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"]
Aug 04 18:13:19.196123 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.196113    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="storage-initializer"
Aug 04 18:13:19.196162 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.196126    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="storage-initializer"
Aug 04 18:13:19.196162 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.196142    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kserve-container"
Aug 04 18:13:19.196162 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.196147    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kserve-container"
Aug 04 18:13:19.196260 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.196164    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kube-rbac-proxy"
Aug 04 18:13:19.196260 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.196170    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kube-rbac-proxy"
Aug 04 18:13:19.196260 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.196219    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kube-rbac-proxy"
Aug 04 18:13:19.196260 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.196231    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f530f5b-5bc2-4f29-a296-2d6b055a30e5" containerName="kserve-container"
Aug 04 18:13:19.201217 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.201197    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:19.203953 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.203935    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-3f1c38-kube-rbac-proxy-sar-config\""
Aug 04 18:13:19.204060 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.203989    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-3f1c38-predictor-serving-cert\""
Aug 04 18:13:19.210776 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.210754    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"]
Aug 04 18:13:19.286888 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.286847    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq7bc\" (UniqueName: \"kubernetes.io/projected/31a81933-171b-4457-afb2-349313ee122b-kube-api-access-tq7bc\") pod \"isvc-primary-3f1c38-predictor-654c9df8cd-nlm44\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") " pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:19.287063 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.286927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31a81933-171b-4457-afb2-349313ee122b-cabundle-cert\") pod \"isvc-primary-3f1c38-predictor-654c9df8cd-nlm44\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") " pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:19.287063 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.286960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31a81933-171b-4457-afb2-349313ee122b-proxy-tls\") pod \"isvc-primary-3f1c38-predictor-654c9df8cd-nlm44\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") " pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:19.287063 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.286997    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31a81933-171b-4457-afb2-349313ee122b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-3f1c38-predictor-654c9df8cd-nlm44\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") " pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:19.287063 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.287023    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31a81933-171b-4457-afb2-349313ee122b-kserve-provision-location\") pod \"isvc-primary-3f1c38-predictor-654c9df8cd-nlm44\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") " pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:19.387927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.387889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31a81933-171b-4457-afb2-349313ee122b-cabundle-cert\") pod \"isvc-primary-3f1c38-predictor-654c9df8cd-nlm44\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") " pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:19.388113 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.387947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31a81933-171b-4457-afb2-349313ee122b-proxy-tls\") pod \"isvc-primary-3f1c38-predictor-654c9df8cd-nlm44\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") " pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:19.388113 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.387987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31a81933-171b-4457-afb2-349313ee122b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-3f1c38-predictor-654c9df8cd-nlm44\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") " pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:19.388113 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.388005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31a81933-171b-4457-afb2-349313ee122b-kserve-provision-location\") pod \"isvc-primary-3f1c38-predictor-654c9df8cd-nlm44\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") " pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:19.388113 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.388026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq7bc\" (UniqueName: \"kubernetes.io/projected/31a81933-171b-4457-afb2-349313ee122b-kube-api-access-tq7bc\") pod \"isvc-primary-3f1c38-predictor-654c9df8cd-nlm44\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") " pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:19.388398 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:13:19.388193    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-serving-cert: secret "isvc-primary-3f1c38-predictor-serving-cert" not found
Aug 04 18:13:19.388398 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:13:19.388257    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31a81933-171b-4457-afb2-349313ee122b-proxy-tls podName:31a81933-171b-4457-afb2-349313ee122b nodeName:}" failed. No retries permitted until 2026-08-04 18:13:19.888237538 +0000 UTC m=+799.178851688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/31a81933-171b-4457-afb2-349313ee122b-proxy-tls") pod "isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" (UID: "31a81933-171b-4457-afb2-349313ee122b") : secret "isvc-primary-3f1c38-predictor-serving-cert" not found
Aug 04 18:13:19.388540 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.388476    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31a81933-171b-4457-afb2-349313ee122b-kserve-provision-location\") pod \"isvc-primary-3f1c38-predictor-654c9df8cd-nlm44\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") " pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:19.388769 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.388748    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31a81933-171b-4457-afb2-349313ee122b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-3f1c38-predictor-654c9df8cd-nlm44\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") " pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:19.388806 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.388749    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31a81933-171b-4457-afb2-349313ee122b-cabundle-cert\") pod \"isvc-primary-3f1c38-predictor-654c9df8cd-nlm44\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") " pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:19.399648 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.399624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq7bc\" (UniqueName: \"kubernetes.io/projected/31a81933-171b-4457-afb2-349313ee122b-kube-api-access-tq7bc\") pod \"isvc-primary-3f1c38-predictor-654c9df8cd-nlm44\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") " pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:19.892958 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.892921    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31a81933-171b-4457-afb2-349313ee122b-proxy-tls\") pod \"isvc-primary-3f1c38-predictor-654c9df8cd-nlm44\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") " pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:19.895326 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:19.895299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31a81933-171b-4457-afb2-349313ee122b-proxy-tls\") pod \"isvc-primary-3f1c38-predictor-654c9df8cd-nlm44\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") " pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:20.112601 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:20.112563    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:20.253016 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:20.252989    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"]
Aug 04 18:13:20.255426 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:13:20.255399    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a81933_171b_4457_afb2_349313ee122b.slice/crio-e8400f31f19b2da589df08f69b658eee7d378773f40d93ccd9f05662427ef679 WatchSource:0}: Error finding container e8400f31f19b2da589df08f69b658eee7d378773f40d93ccd9f05662427ef679: Status 404 returned error can't find the container with id e8400f31f19b2da589df08f69b658eee7d378773f40d93ccd9f05662427ef679
Aug 04 18:13:20.795450 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:20.795412    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" event={"ID":"31a81933-171b-4457-afb2-349313ee122b","Type":"ContainerStarted","Data":"93ed95134de82eb9f970b00fd266a05d7706f2a6c8f0d98de6969c1112964141"}
Aug 04 18:13:20.795611 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:20.795493    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" event={"ID":"31a81933-171b-4457-afb2-349313ee122b","Type":"ContainerStarted","Data":"e8400f31f19b2da589df08f69b658eee7d378773f40d93ccd9f05662427ef679"}
Aug 04 18:13:21.288868 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:21.288828    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Aug 04 18:13:23.297682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:23.297585    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 04 18:13:23.298060 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:23.297896    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:13:24.821916 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:24.821884    2566 generic.go:356] "Generic (PLEG): container finished" podID="31a81933-171b-4457-afb2-349313ee122b" containerID="93ed95134de82eb9f970b00fd266a05d7706f2a6c8f0d98de6969c1112964141" exitCode=0
Aug 04 18:13:24.822285 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:24.821962    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" event={"ID":"31a81933-171b-4457-afb2-349313ee122b","Type":"ContainerDied","Data":"93ed95134de82eb9f970b00fd266a05d7706f2a6c8f0d98de6969c1112964141"}
Aug 04 18:13:25.829510 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:25.829478    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" event={"ID":"31a81933-171b-4457-afb2-349313ee122b","Type":"ContainerStarted","Data":"0af5bd0ab7965211ec6b571d085e601a2ea5924951eb2734ff0d457c7bbd272d"}
Aug 04 18:13:25.829902 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:25.829544    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" event={"ID":"31a81933-171b-4457-afb2-349313ee122b","Type":"ContainerStarted","Data":"9f3c42b705c8d649fa7b7a71770b5d7e1563a687c6e08a95a0e834b3c0c13fc2"}
Aug 04 18:13:25.829902 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:25.829744    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:25.853565 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:25.853503    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" podStartSLOduration=3.14169481 podStartE2EDuration="6.853480362s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.711785552s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:13:25.852683046 +0000 UTC m=+805.143297219" watchObservedRunningTime="2026-08-04 18:13:25.853480362 +0000 UTC m=+805.144094534"
Aug 04 18:13:26.289693 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:26.289597    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Aug 04 18:13:26.834222 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:26.834189    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:26.835446 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:26.835422    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 04 18:13:27.753506 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:27.753465    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Aug 04 18:13:27.754007 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:27.753978    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:13:27.838915 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:27.838877    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 04 18:13:31.289313 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:31.289272    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Aug 04 18:13:32.843987 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:32.843961    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:13:32.844490 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:32.844463    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 04 18:13:33.297600 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:33.297499    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Aug 04 18:13:33.297765 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:33.297649    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:13:33.297839 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:33.297813    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:13:33.297962 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:33.297949    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:13:34.795947 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.795906    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:13:34.876455 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.876420    2566 generic.go:356] "Generic (PLEG): container finished" podID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerID="ed686af051dfa5566a294aac62a61c471342f0f49c5fb62408c87e4ffcb618a5" exitCode=0
Aug 04 18:13:34.876610 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.876491    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"
Aug 04 18:13:34.876610 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.876505    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" event={"ID":"7bec084e-18cb-46f7-8479-e6081c14f0b9","Type":"ContainerDied","Data":"ed686af051dfa5566a294aac62a61c471342f0f49c5fb62408c87e4ffcb618a5"}
Aug 04 18:13:34.876705 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.876621    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh" event={"ID":"7bec084e-18cb-46f7-8479-e6081c14f0b9","Type":"ContainerDied","Data":"b864768c871ff72f79fb01c795f4c1281249c25d3f3fe06ef04d52f409f3f86a"}
Aug 04 18:13:34.876705 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.876685    2566 scope.go:117] "RemoveContainer" containerID="ed686af051dfa5566a294aac62a61c471342f0f49c5fb62408c87e4ffcb618a5"
Aug 04 18:13:34.884859 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.884841    2566 scope.go:117] "RemoveContainer" containerID="fbe3a111ebb34458c4fe914b797e19b70be03eda617bcbbcf50659c79489f89b"
Aug 04 18:13:34.892279 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.892255    2566 scope.go:117] "RemoveContainer" containerID="37be873247c37ba348c02e849fe347cbe38331bd5d73baa6634fc0f91d8b0172"
Aug 04 18:13:34.899653 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.899630    2566 scope.go:117] "RemoveContainer" containerID="b6ed138164ef8826c2f2e75a6662001ec66c4c560021dabc502eb91051004f0d"
Aug 04 18:13:34.910540 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.910523    2566 scope.go:117] "RemoveContainer" containerID="ed686af051dfa5566a294aac62a61c471342f0f49c5fb62408c87e4ffcb618a5"
Aug 04 18:13:34.910839 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:13:34.910815    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed686af051dfa5566a294aac62a61c471342f0f49c5fb62408c87e4ffcb618a5\": container with ID starting with ed686af051dfa5566a294aac62a61c471342f0f49c5fb62408c87e4ffcb618a5 not found: ID does not exist" containerID="ed686af051dfa5566a294aac62a61c471342f0f49c5fb62408c87e4ffcb618a5"
Aug 04 18:13:34.910900 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.910844    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed686af051dfa5566a294aac62a61c471342f0f49c5fb62408c87e4ffcb618a5"} err="failed to get container status \"ed686af051dfa5566a294aac62a61c471342f0f49c5fb62408c87e4ffcb618a5\": rpc error: code = NotFound desc = could not find container \"ed686af051dfa5566a294aac62a61c471342f0f49c5fb62408c87e4ffcb618a5\": container with ID starting with ed686af051dfa5566a294aac62a61c471342f0f49c5fb62408c87e4ffcb618a5 not found: ID does not exist"
Aug 04 18:13:34.910900 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.910863    2566 scope.go:117] "RemoveContainer" containerID="fbe3a111ebb34458c4fe914b797e19b70be03eda617bcbbcf50659c79489f89b"
Aug 04 18:13:34.911119 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:13:34.911104    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe3a111ebb34458c4fe914b797e19b70be03eda617bcbbcf50659c79489f89b\": container with ID starting with fbe3a111ebb34458c4fe914b797e19b70be03eda617bcbbcf50659c79489f89b not found: ID does not exist" containerID="fbe3a111ebb34458c4fe914b797e19b70be03eda617bcbbcf50659c79489f89b"
Aug 04 18:13:34.911170 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.911123    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe3a111ebb34458c4fe914b797e19b70be03eda617bcbbcf50659c79489f89b"} err="failed to get container status \"fbe3a111ebb34458c4fe914b797e19b70be03eda617bcbbcf50659c79489f89b\": rpc error: code = NotFound desc = could not find container \"fbe3a111ebb34458c4fe914b797e19b70be03eda617bcbbcf50659c79489f89b\": container with ID starting with fbe3a111ebb34458c4fe914b797e19b70be03eda617bcbbcf50659c79489f89b not found: ID does not exist"
Aug 04 18:13:34.911170 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.911136    2566 scope.go:117] "RemoveContainer" containerID="37be873247c37ba348c02e849fe347cbe38331bd5d73baa6634fc0f91d8b0172"
Aug 04 18:13:34.911342 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:13:34.911328    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37be873247c37ba348c02e849fe347cbe38331bd5d73baa6634fc0f91d8b0172\": container with ID starting with 37be873247c37ba348c02e849fe347cbe38331bd5d73baa6634fc0f91d8b0172 not found: ID does not exist" containerID="37be873247c37ba348c02e849fe347cbe38331bd5d73baa6634fc0f91d8b0172"
Aug 04 18:13:34.911401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.911345    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37be873247c37ba348c02e849fe347cbe38331bd5d73baa6634fc0f91d8b0172"} err="failed to get container status \"37be873247c37ba348c02e849fe347cbe38331bd5d73baa6634fc0f91d8b0172\": rpc error: code = NotFound desc = could not find container \"37be873247c37ba348c02e849fe347cbe38331bd5d73baa6634fc0f91d8b0172\": container with ID starting with 37be873247c37ba348c02e849fe347cbe38331bd5d73baa6634fc0f91d8b0172 not found: ID does not exist"
Aug 04 18:13:34.911401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.911374    2566 scope.go:117] "RemoveContainer" containerID="b6ed138164ef8826c2f2e75a6662001ec66c4c560021dabc502eb91051004f0d"
Aug 04 18:13:34.911579 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:13:34.911561    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ed138164ef8826c2f2e75a6662001ec66c4c560021dabc502eb91051004f0d\": container with ID starting with b6ed138164ef8826c2f2e75a6662001ec66c4c560021dabc502eb91051004f0d not found: ID does not exist" containerID="b6ed138164ef8826c2f2e75a6662001ec66c4c560021dabc502eb91051004f0d"
Aug 04 18:13:34.911619 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.911585    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ed138164ef8826c2f2e75a6662001ec66c4c560021dabc502eb91051004f0d"} err="failed to get container status \"b6ed138164ef8826c2f2e75a6662001ec66c4c560021dabc502eb91051004f0d\": rpc error: code = NotFound desc = could not find container \"b6ed138164ef8826c2f2e75a6662001ec66c4c560021dabc502eb91051004f0d\": container with ID starting with b6ed138164ef8826c2f2e75a6662001ec66c4c560021dabc502eb91051004f0d not found: ID does not exist"
Aug 04 18:13:34.941927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.941899    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bec084e-18cb-46f7-8479-e6081c14f0b9-kserve-provision-location\") pod \"7bec084e-18cb-46f7-8479-e6081c14f0b9\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") "
Aug 04 18:13:34.942015 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.941970    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bec084e-18cb-46f7-8479-e6081c14f0b9-proxy-tls\") pod \"7bec084e-18cb-46f7-8479-e6081c14f0b9\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") "
Aug 04 18:13:34.942015 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.942000    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw5l2\" (UniqueName: \"kubernetes.io/projected/7bec084e-18cb-46f7-8479-e6081c14f0b9-kube-api-access-tw5l2\") pod \"7bec084e-18cb-46f7-8479-e6081c14f0b9\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") "
Aug 04 18:13:34.942105 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.942034    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bec084e-18cb-46f7-8479-e6081c14f0b9-cabundle-cert\") pod \"7bec084e-18cb-46f7-8479-e6081c14f0b9\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") "
Aug 04 18:13:34.942105 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.942058    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bec084e-18cb-46f7-8479-e6081c14f0b9-kube-rbac-proxy-sar-config\") pod \"7bec084e-18cb-46f7-8479-e6081c14f0b9\" (UID: \"7bec084e-18cb-46f7-8479-e6081c14f0b9\") "
Aug 04 18:13:34.942376 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.942314    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bec084e-18cb-46f7-8479-e6081c14f0b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7bec084e-18cb-46f7-8479-e6081c14f0b9" (UID: "7bec084e-18cb-46f7-8479-e6081c14f0b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:13:34.942462 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.942447    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bec084e-18cb-46f7-8479-e6081c14f0b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7bec084e-18cb-46f7-8479-e6081c14f0b9" (UID: "7bec084e-18cb-46f7-8479-e6081c14f0b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:13:34.942525 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.942502    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bec084e-18cb-46f7-8479-e6081c14f0b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7bec084e-18cb-46f7-8479-e6081c14f0b9" (UID: "7bec084e-18cb-46f7-8479-e6081c14f0b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:13:34.944159 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.944140    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bec084e-18cb-46f7-8479-e6081c14f0b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7bec084e-18cb-46f7-8479-e6081c14f0b9" (UID: "7bec084e-18cb-46f7-8479-e6081c14f0b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:13:34.944225 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:34.944161    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bec084e-18cb-46f7-8479-e6081c14f0b9-kube-api-access-tw5l2" (OuterVolumeSpecName: "kube-api-access-tw5l2") pod "7bec084e-18cb-46f7-8479-e6081c14f0b9" (UID: "7bec084e-18cb-46f7-8479-e6081c14f0b9"). InnerVolumeSpecName "kube-api-access-tw5l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:13:35.042910 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:35.042876    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bec084e-18cb-46f7-8479-e6081c14f0b9-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:13:35.042910 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:35.042907    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw5l2\" (UniqueName: \"kubernetes.io/projected/7bec084e-18cb-46f7-8479-e6081c14f0b9-kube-api-access-tw5l2\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:13:35.042910 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:35.042917    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bec084e-18cb-46f7-8479-e6081c14f0b9-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:13:35.043155 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:35.042926    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bec084e-18cb-46f7-8479-e6081c14f0b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:13:35.043155 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:35.042934    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bec084e-18cb-46f7-8479-e6081c14f0b9-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:13:35.202368 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:35.202320    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"]
Aug 04 18:13:35.207760 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:35.207733    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6b6c6df667-bf7sh"]
Aug 04 18:13:37.194881 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:37.194842    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" path="/var/lib/kubelet/pods/7bec084e-18cb-46f7-8479-e6081c14f0b9/volumes"
Aug 04 18:13:37.753869 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:37.753822    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Aug 04 18:13:37.754232 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:37.754191    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:13:42.844668 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:42.844630    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 04 18:13:47.753039 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:47.752994    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Aug 04 18:13:47.753631 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:47.753585    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:13:52.844487 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:52.844445    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 04 18:13:57.753066 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:57.753024    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Aug 04 18:13:57.753628 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:13:57.753572    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:14:02.845165 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:02.844809    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 04 18:14:07.754005 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:07.753956    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Aug 04 18:14:07.754482 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:07.754465    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:14:12.844782 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:12.844737    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 04 18:14:17.753714 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:17.753677    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Aug 04 18:14:17.754294 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:17.754266    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:14:22.845400 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:22.845342    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 04 18:14:23.189674 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:23.189564    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Aug 04 18:14:23.189829 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:23.189700    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:14:32.844839 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:32.844786    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 04 18:14:33.194534 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:33.194454    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:14:33.194694 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:33.194537    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:14:39.201763 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.201730    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"]
Aug 04 18:14:39.202225 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.202029    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container" containerID="cri-o://2219777eb7f93da198db09aac3f57e8c81174f3dc10ccc748db9af75f753b3b9" gracePeriod=30
Aug 04 18:14:39.202225 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.202099    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent" containerID="cri-o://7649a2364a43cda604be4d413e999fdeb5574cb5906d3ca7be12fc4b667d4cc1" gracePeriod=30
Aug 04 18:14:39.202225 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.202109    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kube-rbac-proxy" containerID="cri-o://10a69171e313da94bc522cee19e008688ce9e49b2e7e4494678ec6d7c1b23f98" gracePeriod=30
Aug 04 18:14:39.230668 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.230643    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"]
Aug 04 18:14:39.231020 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.231008    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="storage-initializer"
Aug 04 18:14:39.231068 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.231022    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="storage-initializer"
Aug 04 18:14:39.231068 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.231031    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kserve-container"
Aug 04 18:14:39.231068 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.231036    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kserve-container"
Aug 04 18:14:39.231068 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.231045    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kube-rbac-proxy"
Aug 04 18:14:39.231068 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.231051    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kube-rbac-proxy"
Aug 04 18:14:39.231068 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.231057    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="agent"
Aug 04 18:14:39.231068 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.231063    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="agent"
Aug 04 18:14:39.231288 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.231129    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kserve-container"
Aug 04 18:14:39.231288 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.231139    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="kube-rbac-proxy"
Aug 04 18:14:39.231288 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.231147    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bec084e-18cb-46f7-8479-e6081c14f0b9" containerName="agent"
Aug 04 18:14:39.234372 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.234343    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:39.236660 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.236636    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Aug 04 18:14:39.236771 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.236640    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Aug 04 18:14:39.245608 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.245585    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"]
Aug 04 18:14:39.329917 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.329886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99d23590-fd22-4658-a22c-0b74447c2a55-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-qxm5z\" (UID: \"99d23590-fd22-4658-a22c-0b74447c2a55\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:39.330057 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.329928    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d23590-fd22-4658-a22c-0b74447c2a55-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-qxm5z\" (UID: \"99d23590-fd22-4658-a22c-0b74447c2a55\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:39.330057 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.329994    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9nkx\" (UniqueName: \"kubernetes.io/projected/99d23590-fd22-4658-a22c-0b74447c2a55-kube-api-access-q9nkx\") pod \"message-dumper-predictor-6575bcfdfd-qxm5z\" (UID: \"99d23590-fd22-4658-a22c-0b74447c2a55\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:39.430827 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.430777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99d23590-fd22-4658-a22c-0b74447c2a55-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-qxm5z\" (UID: \"99d23590-fd22-4658-a22c-0b74447c2a55\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:39.430827 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.430832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d23590-fd22-4658-a22c-0b74447c2a55-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-qxm5z\" (UID: \"99d23590-fd22-4658-a22c-0b74447c2a55\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:39.431077 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.430862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9nkx\" (UniqueName: \"kubernetes.io/projected/99d23590-fd22-4658-a22c-0b74447c2a55-kube-api-access-q9nkx\") pod \"message-dumper-predictor-6575bcfdfd-qxm5z\" (UID: \"99d23590-fd22-4658-a22c-0b74447c2a55\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:39.431077 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:14:39.430994    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Aug 04 18:14:39.431077 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:14:39.431066    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99d23590-fd22-4658-a22c-0b74447c2a55-proxy-tls podName:99d23590-fd22-4658-a22c-0b74447c2a55 nodeName:}" failed. No retries permitted until 2026-08-04 18:14:39.931047324 +0000 UTC m=+879.221661473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/99d23590-fd22-4658-a22c-0b74447c2a55-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-qxm5z" (UID: "99d23590-fd22-4658-a22c-0b74447c2a55") : secret "message-dumper-predictor-serving-cert" not found
Aug 04 18:14:39.431439 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.431423    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99d23590-fd22-4658-a22c-0b74447c2a55-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-qxm5z\" (UID: \"99d23590-fd22-4658-a22c-0b74447c2a55\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:39.442766 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.442737    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9nkx\" (UniqueName: \"kubernetes.io/projected/99d23590-fd22-4658-a22c-0b74447c2a55-kube-api-access-q9nkx\") pod \"message-dumper-predictor-6575bcfdfd-qxm5z\" (UID: \"99d23590-fd22-4658-a22c-0b74447c2a55\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:39.935556 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:39.935515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d23590-fd22-4658-a22c-0b74447c2a55-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-qxm5z\" (UID: \"99d23590-fd22-4658-a22c-0b74447c2a55\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:39.935776 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:14:39.935697    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Aug 04 18:14:39.935851 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:14:39.935782    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99d23590-fd22-4658-a22c-0b74447c2a55-proxy-tls podName:99d23590-fd22-4658-a22c-0b74447c2a55 nodeName:}" failed. No retries permitted until 2026-08-04 18:14:40.935761381 +0000 UTC m=+880.226375543 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/99d23590-fd22-4658-a22c-0b74447c2a55-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-qxm5z" (UID: "99d23590-fd22-4658-a22c-0b74447c2a55") : secret "message-dumper-predictor-serving-cert" not found
Aug 04 18:14:40.187432 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:40.187333    2566 generic.go:356] "Generic (PLEG): container finished" podID="452be1b6-5f5b-479d-a422-45532c22117d" containerID="10a69171e313da94bc522cee19e008688ce9e49b2e7e4494678ec6d7c1b23f98" exitCode=2
Aug 04 18:14:40.187432 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:40.187415    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" event={"ID":"452be1b6-5f5b-479d-a422-45532c22117d","Type":"ContainerDied","Data":"10a69171e313da94bc522cee19e008688ce9e49b2e7e4494678ec6d7c1b23f98"}
Aug 04 18:14:40.944536 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:40.944501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d23590-fd22-4658-a22c-0b74447c2a55-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-qxm5z\" (UID: \"99d23590-fd22-4658-a22c-0b74447c2a55\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:40.946960 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:40.946937    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d23590-fd22-4658-a22c-0b74447c2a55-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-qxm5z\" (UID: \"99d23590-fd22-4658-a22c-0b74447c2a55\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:41.044515 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:41.044484    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:41.180674 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:41.180639    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"]
Aug 04 18:14:41.187261 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:41.187241    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:14:41.194337 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:41.194311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z" event={"ID":"99d23590-fd22-4658-a22c-0b74447c2a55","Type":"ContainerStarted","Data":"155a847fe6cf6a3dda09a74e4ad67e97643d2ed9747bc14c02a622714f73c48c"}
Aug 04 18:14:42.747607 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:42.747501    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Aug 04 18:14:42.845578 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:42.845540    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:14:43.190065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:43.190022    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Aug 04 18:14:43.190349 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:43.190327    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:14:43.210123 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:43.210095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z" event={"ID":"99d23590-fd22-4658-a22c-0b74447c2a55","Type":"ContainerStarted","Data":"16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d"}
Aug 04 18:14:43.210273 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:43.210156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z" event={"ID":"99d23590-fd22-4658-a22c-0b74447c2a55","Type":"ContainerStarted","Data":"bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d"}
Aug 04 18:14:43.210327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:43.210293    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:43.230376 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:43.230310    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z" podStartSLOduration=3.034344293 podStartE2EDuration="4.230293529s" totalImagesPullingTime="1.195949236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:14:43.22974074 +0000 UTC m=+882.520354911" watchObservedRunningTime="2026-08-04 18:14:43.230293529 +0000 UTC m=+882.520907720"
Aug 04 18:14:44.217142 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:44.217050    2566 generic.go:356] "Generic (PLEG): container finished" podID="452be1b6-5f5b-479d-a422-45532c22117d" containerID="2219777eb7f93da198db09aac3f57e8c81174f3dc10ccc748db9af75f753b3b9" exitCode=0
Aug 04 18:14:44.217142 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:44.217123    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" event={"ID":"452be1b6-5f5b-479d-a422-45532c22117d","Type":"ContainerDied","Data":"2219777eb7f93da198db09aac3f57e8c81174f3dc10ccc748db9af75f753b3b9"}
Aug 04 18:14:44.217607 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:44.217413    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:44.219162 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:44.219148    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:47.747429 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:47.747388    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Aug 04 18:14:51.230763 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:51.230733    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:14:52.747707 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:52.747665    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Aug 04 18:14:52.748113 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:52.747792    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:14:53.189564 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:53.189469    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Aug 04 18:14:53.189781 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:53.189753    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:14:57.747575 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:57.747526    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Aug 04 18:14:59.318151 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.318116    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"]
Aug 04 18:14:59.321859 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.321838    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:14:59.324072 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.324053    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Aug 04 18:14:59.324238 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.324219    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Aug 04 18:14:59.331942 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.331923    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"]
Aug 04 18:14:59.417043 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.417010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18e64bad-f6e8-4461-9241-84864c191227-proxy-tls\") pod \"isvc-logger-predictor-6cb98666b8-2xpg4\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:14:59.417223 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.417050    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18e64bad-f6e8-4461-9241-84864c191227-kserve-provision-location\") pod \"isvc-logger-predictor-6cb98666b8-2xpg4\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:14:59.417223 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.417077    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18e64bad-f6e8-4461-9241-84864c191227-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6cb98666b8-2xpg4\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:14:59.417223 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.417096    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bqrk\" (UniqueName: \"kubernetes.io/projected/18e64bad-f6e8-4461-9241-84864c191227-kube-api-access-4bqrk\") pod \"isvc-logger-predictor-6cb98666b8-2xpg4\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:14:59.417223 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.417130    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18e64bad-f6e8-4461-9241-84864c191227-cabundle-cert\") pod \"isvc-logger-predictor-6cb98666b8-2xpg4\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:14:59.517987 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.517954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18e64bad-f6e8-4461-9241-84864c191227-kserve-provision-location\") pod \"isvc-logger-predictor-6cb98666b8-2xpg4\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:14:59.518160 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.517994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18e64bad-f6e8-4461-9241-84864c191227-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6cb98666b8-2xpg4\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:14:59.518160 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.518017    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bqrk\" (UniqueName: \"kubernetes.io/projected/18e64bad-f6e8-4461-9241-84864c191227-kube-api-access-4bqrk\") pod \"isvc-logger-predictor-6cb98666b8-2xpg4\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:14:59.518160 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.518042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18e64bad-f6e8-4461-9241-84864c191227-cabundle-cert\") pod \"isvc-logger-predictor-6cb98666b8-2xpg4\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:14:59.518338 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.518161    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18e64bad-f6e8-4461-9241-84864c191227-proxy-tls\") pod \"isvc-logger-predictor-6cb98666b8-2xpg4\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:14:59.518338 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:14:59.518269    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Aug 04 18:14:59.518478 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:14:59.518390    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e64bad-f6e8-4461-9241-84864c191227-proxy-tls podName:18e64bad-f6e8-4461-9241-84864c191227 nodeName:}" failed. No retries permitted until 2026-08-04 18:15:00.018343634 +0000 UTC m=+899.308957798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/18e64bad-f6e8-4461-9241-84864c191227-proxy-tls") pod "isvc-logger-predictor-6cb98666b8-2xpg4" (UID: "18e64bad-f6e8-4461-9241-84864c191227") : secret "isvc-logger-predictor-serving-cert" not found
Aug 04 18:14:59.518478 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.518464    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18e64bad-f6e8-4461-9241-84864c191227-kserve-provision-location\") pod \"isvc-logger-predictor-6cb98666b8-2xpg4\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:14:59.518795 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.518778    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18e64bad-f6e8-4461-9241-84864c191227-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6cb98666b8-2xpg4\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:14:59.518851 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.518834    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18e64bad-f6e8-4461-9241-84864c191227-cabundle-cert\") pod \"isvc-logger-predictor-6cb98666b8-2xpg4\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:14:59.527229 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:14:59.527204    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bqrk\" (UniqueName: \"kubernetes.io/projected/18e64bad-f6e8-4461-9241-84864c191227-kube-api-access-4bqrk\") pod \"isvc-logger-predictor-6cb98666b8-2xpg4\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:15:00.023790 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:00.023750    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18e64bad-f6e8-4461-9241-84864c191227-proxy-tls\") pod \"isvc-logger-predictor-6cb98666b8-2xpg4\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:15:00.026197 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:00.026171    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18e64bad-f6e8-4461-9241-84864c191227-proxy-tls\") pod \"isvc-logger-predictor-6cb98666b8-2xpg4\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:15:00.233605 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:00.233566    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:15:00.375017 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:00.374984    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"]
Aug 04 18:15:00.376768 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:15:00.376736    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e64bad_f6e8_4461_9241_84864c191227.slice/crio-2fbf0db4eb4ea6280937e4d24fb3036d1eb7e6f12605cff668aa73c901fd2cc6 WatchSource:0}: Error finding container 2fbf0db4eb4ea6280937e4d24fb3036d1eb7e6f12605cff668aa73c901fd2cc6: Status 404 returned error can't find the container with id 2fbf0db4eb4ea6280937e4d24fb3036d1eb7e6f12605cff668aa73c901fd2cc6
Aug 04 18:15:01.302731 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:01.302689    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" event={"ID":"18e64bad-f6e8-4461-9241-84864c191227","Type":"ContainerStarted","Data":"68607f9ee7cf40f2663da7bf76b4c96092aa87a80d3a40005c6555b339de0834"}
Aug 04 18:15:01.302929 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:01.302773    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" event={"ID":"18e64bad-f6e8-4461-9241-84864c191227","Type":"ContainerStarted","Data":"2fbf0db4eb4ea6280937e4d24fb3036d1eb7e6f12605cff668aa73c901fd2cc6"}
Aug 04 18:15:02.748074 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:02.748028    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Aug 04 18:15:03.189837 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:03.189793    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Aug 04 18:15:03.190213 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:03.190191    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:15:03.194191 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:03.194170    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:15:03.194313 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:03.194259    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:15:04.319090 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.319045    2566 generic.go:356] "Generic (PLEG): container finished" podID="18e64bad-f6e8-4461-9241-84864c191227" containerID="68607f9ee7cf40f2663da7bf76b4c96092aa87a80d3a40005c6555b339de0834" exitCode=0
Aug 04 18:15:04.319510 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.319113    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" event={"ID":"18e64bad-f6e8-4461-9241-84864c191227","Type":"ContainerDied","Data":"68607f9ee7cf40f2663da7bf76b4c96092aa87a80d3a40005c6555b339de0834"}
Aug 04 18:15:04.360014 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.359984    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"]
Aug 04 18:15:04.364145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.364125    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.366720 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.366700    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-3f1c38-predictor-serving-cert\""
Aug 04 18:15:04.366720 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.366706    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-3f1c38-kube-rbac-proxy-sar-config\""
Aug 04 18:15:04.366912 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.366728    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-3f1c38-dockercfg-l2xm5\""
Aug 04 18:15:04.366912 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.366706    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-3f1c38\""
Aug 04 18:15:04.376039 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.376017    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"]
Aug 04 18:15:04.465141 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.465118    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-proxy-tls\") pod \"isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") " pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.465245 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.465206    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") " pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.465245 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.465225    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kserve-provision-location\") pod \"isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") " pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.465427 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.465247    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-cabundle-cert\") pod \"isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") " pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.465427 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.465307    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwlc5\" (UniqueName: \"kubernetes.io/projected/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kube-api-access-nwlc5\") pod \"isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") " pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.566248 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.566214    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") " pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.566248 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.566250    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kserve-provision-location\") pod \"isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") " pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.566608 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.566270    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-cabundle-cert\") pod \"isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") " pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.566608 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.566298    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwlc5\" (UniqueName: \"kubernetes.io/projected/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kube-api-access-nwlc5\") pod \"isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") " pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.566608 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.566456    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-proxy-tls\") pod \"isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") " pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.566815 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.566696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kserve-provision-location\") pod \"isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") " pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.567113 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.567090    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-cabundle-cert\") pod \"isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") " pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.567181 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.567116    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") " pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.568958 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.568939    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-proxy-tls\") pod \"isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") " pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.577183 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.577129    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwlc5\" (UniqueName: \"kubernetes.io/projected/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kube-api-access-nwlc5\") pod \"isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") " pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.677436 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.677395    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:04.815847 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:04.815820    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"]
Aug 04 18:15:04.817577 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:15:04.817553    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e72780_3348_4d59_9ac0_93fe9f82b8d9.slice/crio-c363a5a6c08918dfaf9772c4c8a31297460b03acfcce925098338ab2047ef515 WatchSource:0}: Error finding container c363a5a6c08918dfaf9772c4c8a31297460b03acfcce925098338ab2047ef515: Status 404 returned error can't find the container with id c363a5a6c08918dfaf9772c4c8a31297460b03acfcce925098338ab2047ef515
Aug 04 18:15:05.326189 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:05.326156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" event={"ID":"18e64bad-f6e8-4461-9241-84864c191227","Type":"ContainerStarted","Data":"456233c7649cc5a06baf41c4b5f232d33bd18711f4c1b536d928e0a41428e125"}
Aug 04 18:15:05.326686 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:05.326233    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" event={"ID":"18e64bad-f6e8-4461-9241-84864c191227","Type":"ContainerStarted","Data":"e50d2dd1eb203857ca6daf855fe07f964d9ff6baadf75d2758536a33deaa07bf"}
Aug 04 18:15:05.326686 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:05.326274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" event={"ID":"18e64bad-f6e8-4461-9241-84864c191227","Type":"ContainerStarted","Data":"3e73c2c1453c31a5afa10fe26f9c97615024bf2bbeb81f44d2ea66851dc93f25"}
Aug 04 18:15:05.326686 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:05.326545    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:15:05.326686 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:05.326597    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:15:05.327954 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:05.327927    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm" event={"ID":"a2e72780-3348-4d59-9ac0-93fe9f82b8d9","Type":"ContainerStarted","Data":"46bc6c5ca277f9f70e320012af0629e1f3b0f9c42f8a39555460a2cd129c9332"}
Aug 04 18:15:05.328231 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:05.327983    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm" event={"ID":"a2e72780-3348-4d59-9ac0-93fe9f82b8d9","Type":"ContainerStarted","Data":"c363a5a6c08918dfaf9772c4c8a31297460b03acfcce925098338ab2047ef515"}
Aug 04 18:15:05.328284 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:05.328219    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 04 18:15:05.351366 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:05.351290    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podStartSLOduration=2.840235745 podStartE2EDuration="6.351272996s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.511037251s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:15:05.350151294 +0000 UTC m=+904.640765464" watchObservedRunningTime="2026-08-04 18:15:05.351272996 +0000 UTC m=+904.641887179"
Aug 04 18:15:06.334997 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:06.334956    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 04 18:15:06.334997 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:06.334989    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:15:06.336086 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:06.336063    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:15:07.339867 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:07.339825    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 04 18:15:07.340300 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:07.340211    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:15:07.747969 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:07.747879    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Aug 04 18:15:09.352484 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.352453    2566 generic.go:356] "Generic (PLEG): container finished" podID="452be1b6-5f5b-479d-a422-45532c22117d" containerID="7649a2364a43cda604be4d413e999fdeb5574cb5906d3ca7be12fc4b667d4cc1" exitCode=0
Aug 04 18:15:09.352911 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.352509    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" event={"ID":"452be1b6-5f5b-479d-a422-45532c22117d","Type":"ContainerDied","Data":"7649a2364a43cda604be4d413e999fdeb5574cb5906d3ca7be12fc4b667d4cc1"}
Aug 04 18:15:09.352911 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.352584    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg" event={"ID":"452be1b6-5f5b-479d-a422-45532c22117d","Type":"ContainerDied","Data":"20d9403367969e88d7410a3788bd80600e5f66574a3cd8a11de4f917efc3590c"}
Aug 04 18:15:09.352911 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.352609    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20d9403367969e88d7410a3788bd80600e5f66574a3cd8a11de4f917efc3590c"
Aug 04 18:15:09.360198 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.360176    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:15:09.517381 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.517257    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/452be1b6-5f5b-479d-a422-45532c22117d-kube-rbac-proxy-sar-config\") pod \"452be1b6-5f5b-479d-a422-45532c22117d\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") "
Aug 04 18:15:09.517381 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.517313    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/452be1b6-5f5b-479d-a422-45532c22117d-proxy-tls\") pod \"452be1b6-5f5b-479d-a422-45532c22117d\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") "
Aug 04 18:15:09.517381 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.517338    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btvcm\" (UniqueName: \"kubernetes.io/projected/452be1b6-5f5b-479d-a422-45532c22117d-kube-api-access-btvcm\") pod \"452be1b6-5f5b-479d-a422-45532c22117d\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") "
Aug 04 18:15:09.517629 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.517407    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/452be1b6-5f5b-479d-a422-45532c22117d-kserve-provision-location\") pod \"452be1b6-5f5b-479d-a422-45532c22117d\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") "
Aug 04 18:15:09.517629 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.517462    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/452be1b6-5f5b-479d-a422-45532c22117d-cabundle-cert\") pod \"452be1b6-5f5b-479d-a422-45532c22117d\" (UID: \"452be1b6-5f5b-479d-a422-45532c22117d\") "
Aug 04 18:15:09.517741 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.517713    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452be1b6-5f5b-479d-a422-45532c22117d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "452be1b6-5f5b-479d-a422-45532c22117d" (UID: "452be1b6-5f5b-479d-a422-45532c22117d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:09.517912 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.517883    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/452be1b6-5f5b-479d-a422-45532c22117d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "452be1b6-5f5b-479d-a422-45532c22117d" (UID: "452be1b6-5f5b-479d-a422-45532c22117d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:15:09.518009 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.517968    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/452be1b6-5f5b-479d-a422-45532c22117d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "452be1b6-5f5b-479d-a422-45532c22117d" (UID: "452be1b6-5f5b-479d-a422-45532c22117d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:15:09.519547 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.519517    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452be1b6-5f5b-479d-a422-45532c22117d-kube-api-access-btvcm" (OuterVolumeSpecName: "kube-api-access-btvcm") pod "452be1b6-5f5b-479d-a422-45532c22117d" (UID: "452be1b6-5f5b-479d-a422-45532c22117d"). InnerVolumeSpecName "kube-api-access-btvcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:15:09.519665 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.519591    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/452be1b6-5f5b-479d-a422-45532c22117d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "452be1b6-5f5b-479d-a422-45532c22117d" (UID: "452be1b6-5f5b-479d-a422-45532c22117d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:15:09.618988 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.618954    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/452be1b6-5f5b-479d-a422-45532c22117d-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:09.618988 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.618990    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btvcm\" (UniqueName: \"kubernetes.io/projected/452be1b6-5f5b-479d-a422-45532c22117d-kube-api-access-btvcm\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:09.619183 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.619005    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/452be1b6-5f5b-479d-a422-45532c22117d-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:09.619183 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.619020    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/452be1b6-5f5b-479d-a422-45532c22117d-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:09.619183 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:09.619034    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/452be1b6-5f5b-479d-a422-45532c22117d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:10.357179 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:10.357147    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"
Aug 04 18:15:10.385779 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:10.385742    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"]
Aug 04 18:15:10.389120 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:10.389094    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59d7c547c8-mfrhg"]
Aug 04 18:15:11.195511 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:11.195480    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="452be1b6-5f5b-479d-a422-45532c22117d" path="/var/lib/kubelet/pods/452be1b6-5f5b-479d-a422-45532c22117d/volumes"
Aug 04 18:15:11.363650 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:11.363621    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm_a2e72780-3348-4d59-9ac0-93fe9f82b8d9/storage-initializer/0.log"
Aug 04 18:15:11.364031 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:11.363662    2566 generic.go:356] "Generic (PLEG): container finished" podID="a2e72780-3348-4d59-9ac0-93fe9f82b8d9" containerID="46bc6c5ca277f9f70e320012af0629e1f3b0f9c42f8a39555460a2cd129c9332" exitCode=1
Aug 04 18:15:11.364031 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:11.363733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm" event={"ID":"a2e72780-3348-4d59-9ac0-93fe9f82b8d9","Type":"ContainerDied","Data":"46bc6c5ca277f9f70e320012af0629e1f3b0f9c42f8a39555460a2cd129c9332"}
Aug 04 18:15:12.344435 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:12.344405    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:15:12.345028 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:12.344997    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 04 18:15:12.345312 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:12.345284    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:15:12.372686 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:12.372658    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm_a2e72780-3348-4d59-9ac0-93fe9f82b8d9/storage-initializer/0.log"
Aug 04 18:15:12.373086 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:12.372718    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm" event={"ID":"a2e72780-3348-4d59-9ac0-93fe9f82b8d9","Type":"ContainerStarted","Data":"e4745834fd0b781f04be2f64b555ca2002d246b949627de61f79943d8a42d4f5"}
Aug 04 18:15:15.389699 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:15.389624    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm_a2e72780-3348-4d59-9ac0-93fe9f82b8d9/storage-initializer/1.log"
Aug 04 18:15:15.390081 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:15.390015    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm_a2e72780-3348-4d59-9ac0-93fe9f82b8d9/storage-initializer/0.log"
Aug 04 18:15:15.390081 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:15.390052    2566 generic.go:356] "Generic (PLEG): container finished" podID="a2e72780-3348-4d59-9ac0-93fe9f82b8d9" containerID="e4745834fd0b781f04be2f64b555ca2002d246b949627de61f79943d8a42d4f5" exitCode=1
Aug 04 18:15:15.390170 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:15.390135    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm" event={"ID":"a2e72780-3348-4d59-9ac0-93fe9f82b8d9","Type":"ContainerDied","Data":"e4745834fd0b781f04be2f64b555ca2002d246b949627de61f79943d8a42d4f5"}
Aug 04 18:15:15.390212 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:15.390201    2566 scope.go:117] "RemoveContainer" containerID="46bc6c5ca277f9f70e320012af0629e1f3b0f9c42f8a39555460a2cd129c9332"
Aug 04 18:15:15.390619 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:15.390590    2566 scope.go:117] "RemoveContainer" containerID="46bc6c5ca277f9f70e320012af0629e1f3b0f9c42f8a39555460a2cd129c9332"
Aug 04 18:15:15.403526 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:15:15.403504    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm_kserve-ci-e2e-test(a2e72780-3348-4d59-9ac0-93fe9f82b8d9)\"" pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm" podUID="a2e72780-3348-4d59-9ac0-93fe9f82b8d9"
Aug 04 18:15:15.403628 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:15:15.403599    2566 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm_kserve-ci-e2e-test_a2e72780-3348-4d59-9ac0-93fe9f82b8d9_0 in pod sandbox c363a5a6c08918dfaf9772c4c8a31297460b03acfcce925098338ab2047ef515 from index: no such id: '46bc6c5ca277f9f70e320012af0629e1f3b0f9c42f8a39555460a2cd129c9332'" containerID="46bc6c5ca277f9f70e320012af0629e1f3b0f9c42f8a39555460a2cd129c9332"
Aug 04 18:15:15.403679 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:15.403639    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bc6c5ca277f9f70e320012af0629e1f3b0f9c42f8a39555460a2cd129c9332"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm_kserve-ci-e2e-test_a2e72780-3348-4d59-9ac0-93fe9f82b8d9_0 in pod sandbox c363a5a6c08918dfaf9772c4c8a31297460b03acfcce925098338ab2047ef515 from index: no such id: '46bc6c5ca277f9f70e320012af0629e1f3b0f9c42f8a39555460a2cd129c9332'"
Aug 04 18:15:16.397519 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:16.397493    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm_a2e72780-3348-4d59-9ac0-93fe9f82b8d9/storage-initializer/1.log"
Aug 04 18:15:22.345845 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.345794    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 04 18:15:22.346279 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.346190    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:15:22.431885 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.431848    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"]
Aug 04 18:15:22.480026 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.479992    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"]
Aug 04 18:15:22.480255 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.480229    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kserve-container" containerID="cri-o://9f3c42b705c8d649fa7b7a71770b5d7e1563a687c6e08a95a0e834b3c0c13fc2" gracePeriod=30
Aug 04 18:15:22.480321 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.480280    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kube-rbac-proxy" containerID="cri-o://0af5bd0ab7965211ec6b571d085e601a2ea5924951eb2734ff0d457c7bbd272d" gracePeriod=30
Aug 04 18:15:22.602163 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.602106    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm_a2e72780-3348-4d59-9ac0-93fe9f82b8d9/storage-initializer/1.log"
Aug 04 18:15:22.604441 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.604422    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:22.640832 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.640798    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-cabundle-cert\") pod \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") "
Aug 04 18:15:22.640993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.640839    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwlc5\" (UniqueName: \"kubernetes.io/projected/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kube-api-access-nwlc5\") pod \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") "
Aug 04 18:15:22.640993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.640867    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kube-rbac-proxy-sar-config\") pod \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") "
Aug 04 18:15:22.640993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.640925    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-proxy-tls\") pod \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") "
Aug 04 18:15:22.640993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.640966    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kserve-provision-location\") pod \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\" (UID: \"a2e72780-3348-4d59-9ac0-93fe9f82b8d9\") "
Aug 04 18:15:22.641252 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.641222    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2e72780-3348-4d59-9ac0-93fe9f82b8d9" (UID: "a2e72780-3348-4d59-9ac0-93fe9f82b8d9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:15:22.641315 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.641262    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2e72780-3348-4d59-9ac0-93fe9f82b8d9" (UID: "a2e72780-3348-4d59-9ac0-93fe9f82b8d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:22.641315 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.641287    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a2e72780-3348-4d59-9ac0-93fe9f82b8d9" (UID: "a2e72780-3348-4d59-9ac0-93fe9f82b8d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:15:22.643037 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.643012    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2e72780-3348-4d59-9ac0-93fe9f82b8d9" (UID: "a2e72780-3348-4d59-9ac0-93fe9f82b8d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:15:22.643037 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.643034    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kube-api-access-nwlc5" (OuterVolumeSpecName: "kube-api-access-nwlc5") pod "a2e72780-3348-4d59-9ac0-93fe9f82b8d9" (UID: "a2e72780-3348-4d59-9ac0-93fe9f82b8d9"). InnerVolumeSpecName "kube-api-access-nwlc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:15:22.742591 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.742548    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:22.742591 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.742584    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:22.742591 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.742597    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:22.742847 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.742608    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwlc5\" (UniqueName: \"kubernetes.io/projected/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kube-api-access-nwlc5\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:22.742847 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.742620    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2e72780-3348-4d59-9ac0-93fe9f82b8d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:22.839299 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.839259    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Aug 04 18:15:22.845081 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:22.845045    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Aug 04 18:15:23.433628 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:23.433596    2566 generic.go:356] "Generic (PLEG): container finished" podID="31a81933-171b-4457-afb2-349313ee122b" containerID="0af5bd0ab7965211ec6b571d085e601a2ea5924951eb2734ff0d457c7bbd272d" exitCode=2
Aug 04 18:15:23.434005 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:23.433674    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" event={"ID":"31a81933-171b-4457-afb2-349313ee122b","Type":"ContainerDied","Data":"0af5bd0ab7965211ec6b571d085e601a2ea5924951eb2734ff0d457c7bbd272d"}
Aug 04 18:15:23.434879 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:23.434861    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm_a2e72780-3348-4d59-9ac0-93fe9f82b8d9/storage-initializer/1.log"
Aug 04 18:15:23.435011 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:23.434928    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm" event={"ID":"a2e72780-3348-4d59-9ac0-93fe9f82b8d9","Type":"ContainerDied","Data":"c363a5a6c08918dfaf9772c4c8a31297460b03acfcce925098338ab2047ef515"}
Aug 04 18:15:23.435011 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:23.434949    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"
Aug 04 18:15:23.435011 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:23.434990    2566 scope.go:117] "RemoveContainer" containerID="e4745834fd0b781f04be2f64b555ca2002d246b949627de61f79943d8a42d4f5"
Aug 04 18:15:23.470926 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:23.470858    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"]
Aug 04 18:15:23.475514 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:23.475485    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-3f1c38-predictor-8d7644d6f-db6qm"]
Aug 04 18:15:25.194690 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:25.194653    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2e72780-3348-4d59-9ac0-93fe9f82b8d9" path="/var/lib/kubelet/pods/a2e72780-3348-4d59-9ac0-93fe9f82b8d9/volumes"
Aug 04 18:15:27.327586 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.327561    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:15:27.383046 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.383013    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31a81933-171b-4457-afb2-349313ee122b-kube-rbac-proxy-sar-config\") pod \"31a81933-171b-4457-afb2-349313ee122b\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") "
Aug 04 18:15:27.383249 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.383065    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31a81933-171b-4457-afb2-349313ee122b-proxy-tls\") pod \"31a81933-171b-4457-afb2-349313ee122b\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") "
Aug 04 18:15:27.383249 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.383090    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq7bc\" (UniqueName: \"kubernetes.io/projected/31a81933-171b-4457-afb2-349313ee122b-kube-api-access-tq7bc\") pod \"31a81933-171b-4457-afb2-349313ee122b\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") "
Aug 04 18:15:27.383249 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.383117    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31a81933-171b-4457-afb2-349313ee122b-cabundle-cert\") pod \"31a81933-171b-4457-afb2-349313ee122b\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") "
Aug 04 18:15:27.383249 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.383177    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31a81933-171b-4457-afb2-349313ee122b-kserve-provision-location\") pod \"31a81933-171b-4457-afb2-349313ee122b\" (UID: \"31a81933-171b-4457-afb2-349313ee122b\") "
Aug 04 18:15:27.383518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.383495    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a81933-171b-4457-afb2-349313ee122b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "31a81933-171b-4457-afb2-349313ee122b" (UID: "31a81933-171b-4457-afb2-349313ee122b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:15:27.383629 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.383489    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a81933-171b-4457-afb2-349313ee122b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "31a81933-171b-4457-afb2-349313ee122b" (UID: "31a81933-171b-4457-afb2-349313ee122b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:15:27.383679 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.383642    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31a81933-171b-4457-afb2-349313ee122b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31a81933-171b-4457-afb2-349313ee122b" (UID: "31a81933-171b-4457-afb2-349313ee122b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:27.385170 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.385148    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a81933-171b-4457-afb2-349313ee122b-kube-api-access-tq7bc" (OuterVolumeSpecName: "kube-api-access-tq7bc") pod "31a81933-171b-4457-afb2-349313ee122b" (UID: "31a81933-171b-4457-afb2-349313ee122b"). InnerVolumeSpecName "kube-api-access-tq7bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:15:27.385334 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.385312    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31a81933-171b-4457-afb2-349313ee122b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31a81933-171b-4457-afb2-349313ee122b" (UID: "31a81933-171b-4457-afb2-349313ee122b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:15:27.458813 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.458729    2566 generic.go:356] "Generic (PLEG): container finished" podID="31a81933-171b-4457-afb2-349313ee122b" containerID="9f3c42b705c8d649fa7b7a71770b5d7e1563a687c6e08a95a0e834b3c0c13fc2" exitCode=0
Aug 04 18:15:27.458813 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.458794    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"
Aug 04 18:15:27.458987 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.458806    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" event={"ID":"31a81933-171b-4457-afb2-349313ee122b","Type":"ContainerDied","Data":"9f3c42b705c8d649fa7b7a71770b5d7e1563a687c6e08a95a0e834b3c0c13fc2"}
Aug 04 18:15:27.458987 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.458876    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44" event={"ID":"31a81933-171b-4457-afb2-349313ee122b","Type":"ContainerDied","Data":"e8400f31f19b2da589df08f69b658eee7d378773f40d93ccd9f05662427ef679"}
Aug 04 18:15:27.458987 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.458914    2566 scope.go:117] "RemoveContainer" containerID="0af5bd0ab7965211ec6b571d085e601a2ea5924951eb2734ff0d457c7bbd272d"
Aug 04 18:15:27.467678 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.467660    2566 scope.go:117] "RemoveContainer" containerID="9f3c42b705c8d649fa7b7a71770b5d7e1563a687c6e08a95a0e834b3c0c13fc2"
Aug 04 18:15:27.476416 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.476400    2566 scope.go:117] "RemoveContainer" containerID="93ed95134de82eb9f970b00fd266a05d7706f2a6c8f0d98de6969c1112964141"
Aug 04 18:15:27.484054 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.484033    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31a81933-171b-4457-afb2-349313ee122b-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:27.484054 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.484054    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31a81933-171b-4457-afb2-349313ee122b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:27.484190 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.484066    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31a81933-171b-4457-afb2-349313ee122b-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:27.484190 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.484075    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq7bc\" (UniqueName: \"kubernetes.io/projected/31a81933-171b-4457-afb2-349313ee122b-kube-api-access-tq7bc\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:27.484190 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.484076    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"]
Aug 04 18:15:27.484190 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.484085    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31a81933-171b-4457-afb2-349313ee122b-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:27.485497 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.485478    2566 scope.go:117] "RemoveContainer" containerID="0af5bd0ab7965211ec6b571d085e601a2ea5924951eb2734ff0d457c7bbd272d"
Aug 04 18:15:27.485752 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:15:27.485734    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af5bd0ab7965211ec6b571d085e601a2ea5924951eb2734ff0d457c7bbd272d\": container with ID starting with 0af5bd0ab7965211ec6b571d085e601a2ea5924951eb2734ff0d457c7bbd272d not found: ID does not exist" containerID="0af5bd0ab7965211ec6b571d085e601a2ea5924951eb2734ff0d457c7bbd272d"
Aug 04 18:15:27.485815 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.485766    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af5bd0ab7965211ec6b571d085e601a2ea5924951eb2734ff0d457c7bbd272d"} err="failed to get container status \"0af5bd0ab7965211ec6b571d085e601a2ea5924951eb2734ff0d457c7bbd272d\": rpc error: code = NotFound desc = could not find container \"0af5bd0ab7965211ec6b571d085e601a2ea5924951eb2734ff0d457c7bbd272d\": container with ID starting with 0af5bd0ab7965211ec6b571d085e601a2ea5924951eb2734ff0d457c7bbd272d not found: ID does not exist"
Aug 04 18:15:27.485815 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.485787    2566 scope.go:117] "RemoveContainer" containerID="9f3c42b705c8d649fa7b7a71770b5d7e1563a687c6e08a95a0e834b3c0c13fc2"
Aug 04 18:15:27.486006 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:15:27.485989    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3c42b705c8d649fa7b7a71770b5d7e1563a687c6e08a95a0e834b3c0c13fc2\": container with ID starting with 9f3c42b705c8d649fa7b7a71770b5d7e1563a687c6e08a95a0e834b3c0c13fc2 not found: ID does not exist" containerID="9f3c42b705c8d649fa7b7a71770b5d7e1563a687c6e08a95a0e834b3c0c13fc2"
Aug 04 18:15:27.486065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.486012    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3c42b705c8d649fa7b7a71770b5d7e1563a687c6e08a95a0e834b3c0c13fc2"} err="failed to get container status \"9f3c42b705c8d649fa7b7a71770b5d7e1563a687c6e08a95a0e834b3c0c13fc2\": rpc error: code = NotFound desc = could not find container \"9f3c42b705c8d649fa7b7a71770b5d7e1563a687c6e08a95a0e834b3c0c13fc2\": container with ID starting with 9f3c42b705c8d649fa7b7a71770b5d7e1563a687c6e08a95a0e834b3c0c13fc2 not found: ID does not exist"
Aug 04 18:15:27.486065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.486029    2566 scope.go:117] "RemoveContainer" containerID="93ed95134de82eb9f970b00fd266a05d7706f2a6c8f0d98de6969c1112964141"
Aug 04 18:15:27.486248 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:15:27.486232    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93ed95134de82eb9f970b00fd266a05d7706f2a6c8f0d98de6969c1112964141\": container with ID starting with 93ed95134de82eb9f970b00fd266a05d7706f2a6c8f0d98de6969c1112964141 not found: ID does not exist" containerID="93ed95134de82eb9f970b00fd266a05d7706f2a6c8f0d98de6969c1112964141"
Aug 04 18:15:27.486295 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.486252    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ed95134de82eb9f970b00fd266a05d7706f2a6c8f0d98de6969c1112964141"} err="failed to get container status \"93ed95134de82eb9f970b00fd266a05d7706f2a6c8f0d98de6969c1112964141\": rpc error: code = NotFound desc = could not find container \"93ed95134de82eb9f970b00fd266a05d7706f2a6c8f0d98de6969c1112964141\": container with ID starting with 93ed95134de82eb9f970b00fd266a05d7706f2a6c8f0d98de6969c1112964141 not found: ID does not exist"
Aug 04 18:15:27.488926 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.488904    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-3f1c38-predictor-654c9df8cd-nlm44"]
Aug 04 18:15:27.655431 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655399    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"]
Aug 04 18:15:27.655811 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655798    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="storage-initializer"
Aug 04 18:15:27.655854 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655813    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="storage-initializer"
Aug 04 18:15:27.655854 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655827    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent"
Aug 04 18:15:27.655854 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655833    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent"
Aug 04 18:15:27.655854 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655841    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container"
Aug 04 18:15:27.655854 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655847    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container"
Aug 04 18:15:27.655998 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655855    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2e72780-3348-4d59-9ac0-93fe9f82b8d9" containerName="storage-initializer"
Aug 04 18:15:27.655998 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655864    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2e72780-3348-4d59-9ac0-93fe9f82b8d9" containerName="storage-initializer"
Aug 04 18:15:27.655998 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655874    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2e72780-3348-4d59-9ac0-93fe9f82b8d9" containerName="storage-initializer"
Aug 04 18:15:27.655998 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655879    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2e72780-3348-4d59-9ac0-93fe9f82b8d9" containerName="storage-initializer"
Aug 04 18:15:27.655998 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655891    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kube-rbac-proxy"
Aug 04 18:15:27.655998 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655896    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kube-rbac-proxy"
Aug 04 18:15:27.655998 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655903    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kserve-container"
Aug 04 18:15:27.655998 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655908    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kserve-container"
Aug 04 18:15:27.655998 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655916    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="storage-initializer"
Aug 04 18:15:27.655998 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655921    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="storage-initializer"
Aug 04 18:15:27.655998 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655927    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kube-rbac-proxy"
Aug 04 18:15:27.655998 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.655932    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kube-rbac-proxy"
Aug 04 18:15:27.656373 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.656009    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kube-rbac-proxy"
Aug 04 18:15:27.656373 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.656019    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2e72780-3348-4d59-9ac0-93fe9f82b8d9" containerName="storage-initializer"
Aug 04 18:15:27.656373 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.656031    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kube-rbac-proxy"
Aug 04 18:15:27.656373 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.656037    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="agent"
Aug 04 18:15:27.656373 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.656043    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="31a81933-171b-4457-afb2-349313ee122b" containerName="kserve-container"
Aug 04 18:15:27.656373 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.656050    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="452be1b6-5f5b-479d-a422-45532c22117d" containerName="kserve-container"
Aug 04 18:15:27.656373 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.656056    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2e72780-3348-4d59-9ac0-93fe9f82b8d9" containerName="storage-initializer"
Aug 04 18:15:27.660964 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.660942    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:27.663409 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.663386    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-f03bc8-kube-rbac-proxy-sar-config\""
Aug 04 18:15:27.663409 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.663407    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-f03bc8-predictor-serving-cert\""
Aug 04 18:15:27.663576 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.663414    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-f03bc8-dockercfg-fjzhp\""
Aug 04 18:15:27.663576 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.663457    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-f03bc8\""
Aug 04 18:15:27.669729 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.669710    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"]
Aug 04 18:15:27.685617 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.685593    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") " pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:27.685741 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.685646    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kserve-provision-location\") pod \"isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") " pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:27.685741 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.685689    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbhzf\" (UniqueName: \"kubernetes.io/projected/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kube-api-access-sbhzf\") pod \"isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") " pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:27.685741 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.685711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-proxy-tls\") pod \"isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") " pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:27.685741 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.685734    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-cabundle-cert\") pod \"isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") " pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:27.786844 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.786744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-cabundle-cert\") pod \"isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") " pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:27.786844 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.786819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") " pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:27.787093 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.786860    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kserve-provision-location\") pod \"isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") " pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:27.787093 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.786885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbhzf\" (UniqueName: \"kubernetes.io/projected/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kube-api-access-sbhzf\") pod \"isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") " pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:27.787093 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.786907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-proxy-tls\") pod \"isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") " pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:27.787313 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.787290    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kserve-provision-location\") pod \"isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") " pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:27.787448 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.787431    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-cabundle-cert\") pod \"isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") " pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:27.787580 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.787561    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") " pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:27.789321 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.789304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-proxy-tls\") pod \"isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") " pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:27.798526 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.798502    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbhzf\" (UniqueName: \"kubernetes.io/projected/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kube-api-access-sbhzf\") pod \"isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") " pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:27.972201 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:27.972154    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:28.105461 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:28.105429    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"]
Aug 04 18:15:28.107018 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:15:28.106991    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9564eb1f_1ef8_48ac_a6b3_a804d1b01c2a.slice/crio-0e5b06aa82c3dff148c3a499fbdbe5204ea0f2bf3770db3e2f4ee52cdb6abab7 WatchSource:0}: Error finding container 0e5b06aa82c3dff148c3a499fbdbe5204ea0f2bf3770db3e2f4ee52cdb6abab7: Status 404 returned error can't find the container with id 0e5b06aa82c3dff148c3a499fbdbe5204ea0f2bf3770db3e2f4ee52cdb6abab7
Aug 04 18:15:28.467217 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:28.467126    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs" event={"ID":"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a","Type":"ContainerStarted","Data":"d38a6b1a82c2ef267fc3a93361d63c4d6ec521ab311370796fecaeacc9a3c3ff"}
Aug 04 18:15:28.467217 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:28.467182    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs" event={"ID":"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a","Type":"ContainerStarted","Data":"0e5b06aa82c3dff148c3a499fbdbe5204ea0f2bf3770db3e2f4ee52cdb6abab7"}
Aug 04 18:15:29.194737 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:29.194702    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31a81933-171b-4457-afb2-349313ee122b" path="/var/lib/kubelet/pods/31a81933-171b-4457-afb2-349313ee122b/volumes"
Aug 04 18:15:32.344916 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:32.344867    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 04 18:15:32.345436 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:32.345327    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:15:35.504687 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:35.504657    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs_9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a/storage-initializer/0.log"
Aug 04 18:15:35.505055 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:35.504700    2566 generic.go:356] "Generic (PLEG): container finished" podID="9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a" containerID="d38a6b1a82c2ef267fc3a93361d63c4d6ec521ab311370796fecaeacc9a3c3ff" exitCode=1
Aug 04 18:15:35.505055 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:35.504744    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs" event={"ID":"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a","Type":"ContainerDied","Data":"d38a6b1a82c2ef267fc3a93361d63c4d6ec521ab311370796fecaeacc9a3c3ff"}
Aug 04 18:15:36.511761 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:36.511735    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs_9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a/storage-initializer/0.log"
Aug 04 18:15:36.512134 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:36.511852    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs" event={"ID":"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a","Type":"ContainerStarted","Data":"3d1a03e8665592e36a1f2e2372b1c7c98aed90c7d32a0394549f14f2904711bd"}
Aug 04 18:15:37.638954 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:37.638920    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"]
Aug 04 18:15:37.639438 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:37.639140    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs" podUID="9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a" containerName="storage-initializer" containerID="cri-o://3d1a03e8665592e36a1f2e2372b1c7c98aed90c7d32a0394549f14f2904711bd" gracePeriod=30
Aug 04 18:15:41.873013 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:41.872990    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs_9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a/storage-initializer/1.log"
Aug 04 18:15:41.873422 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:41.873407    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs_9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a/storage-initializer/0.log"
Aug 04 18:15:41.875616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:41.875601    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:42.017828 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.017731    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbhzf\" (UniqueName: \"kubernetes.io/projected/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kube-api-access-sbhzf\") pod \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") "
Aug 04 18:15:42.018006 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.017869    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kube-rbac-proxy-sar-config\") pod \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") "
Aug 04 18:15:42.018006 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.017907    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kserve-provision-location\") pod \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") "
Aug 04 18:15:42.018006 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.017968    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-proxy-tls\") pod \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") "
Aug 04 18:15:42.018174 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.018007    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-cabundle-cert\") pod \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\" (UID: \"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a\") "
Aug 04 18:15:42.018255 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.018224    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a" (UID: "9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:42.018333 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.018308    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a" (UID: "9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:15:42.018438 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.018407    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a" (UID: "9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:15:42.019903 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.019871    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kube-api-access-sbhzf" (OuterVolumeSpecName: "kube-api-access-sbhzf") pod "9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a" (UID: "9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a"). InnerVolumeSpecName "kube-api-access-sbhzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:15:42.019903 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.019898    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a" (UID: "9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:15:42.119749 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.119712    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:42.119749 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.119744    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbhzf\" (UniqueName: \"kubernetes.io/projected/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kube-api-access-sbhzf\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:42.119749 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.119755    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:42.120027 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.119763    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:42.120027 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.119773    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:15:42.344909 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.344868    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 04 18:15:42.345314 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.345287    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:15:42.542286 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.542262    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs_9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a/storage-initializer/1.log"
Aug 04 18:15:42.542632 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.542619    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs_9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a/storage-initializer/0.log"
Aug 04 18:15:42.542695 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.542652    2566 generic.go:356] "Generic (PLEG): container finished" podID="9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a" containerID="3d1a03e8665592e36a1f2e2372b1c7c98aed90c7d32a0394549f14f2904711bd" exitCode=1
Aug 04 18:15:42.542733 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.542719    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"
Aug 04 18:15:42.542774 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.542743    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs" event={"ID":"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a","Type":"ContainerDied","Data":"3d1a03e8665592e36a1f2e2372b1c7c98aed90c7d32a0394549f14f2904711bd"}
Aug 04 18:15:42.542818 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.542807    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs" event={"ID":"9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a","Type":"ContainerDied","Data":"0e5b06aa82c3dff148c3a499fbdbe5204ea0f2bf3770db3e2f4ee52cdb6abab7"}
Aug 04 18:15:42.542859 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.542837    2566 scope.go:117] "RemoveContainer" containerID="3d1a03e8665592e36a1f2e2372b1c7c98aed90c7d32a0394549f14f2904711bd"
Aug 04 18:15:42.553523 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.553504    2566 scope.go:117] "RemoveContainer" containerID="d38a6b1a82c2ef267fc3a93361d63c4d6ec521ab311370796fecaeacc9a3c3ff"
Aug 04 18:15:42.561648 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.561631    2566 scope.go:117] "RemoveContainer" containerID="3d1a03e8665592e36a1f2e2372b1c7c98aed90c7d32a0394549f14f2904711bd"
Aug 04 18:15:42.561944 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:15:42.561923    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d1a03e8665592e36a1f2e2372b1c7c98aed90c7d32a0394549f14f2904711bd\": container with ID starting with 3d1a03e8665592e36a1f2e2372b1c7c98aed90c7d32a0394549f14f2904711bd not found: ID does not exist" containerID="3d1a03e8665592e36a1f2e2372b1c7c98aed90c7d32a0394549f14f2904711bd"
Aug 04 18:15:42.562025 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.561952    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1a03e8665592e36a1f2e2372b1c7c98aed90c7d32a0394549f14f2904711bd"} err="failed to get container status \"3d1a03e8665592e36a1f2e2372b1c7c98aed90c7d32a0394549f14f2904711bd\": rpc error: code = NotFound desc = could not find container \"3d1a03e8665592e36a1f2e2372b1c7c98aed90c7d32a0394549f14f2904711bd\": container with ID starting with 3d1a03e8665592e36a1f2e2372b1c7c98aed90c7d32a0394549f14f2904711bd not found: ID does not exist"
Aug 04 18:15:42.562025 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.561969    2566 scope.go:117] "RemoveContainer" containerID="d38a6b1a82c2ef267fc3a93361d63c4d6ec521ab311370796fecaeacc9a3c3ff"
Aug 04 18:15:42.562221 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:15:42.562205    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d38a6b1a82c2ef267fc3a93361d63c4d6ec521ab311370796fecaeacc9a3c3ff\": container with ID starting with d38a6b1a82c2ef267fc3a93361d63c4d6ec521ab311370796fecaeacc9a3c3ff not found: ID does not exist" containerID="d38a6b1a82c2ef267fc3a93361d63c4d6ec521ab311370796fecaeacc9a3c3ff"
Aug 04 18:15:42.562261 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.562228    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38a6b1a82c2ef267fc3a93361d63c4d6ec521ab311370796fecaeacc9a3c3ff"} err="failed to get container status \"d38a6b1a82c2ef267fc3a93361d63c4d6ec521ab311370796fecaeacc9a3c3ff\": rpc error: code = NotFound desc = could not find container \"d38a6b1a82c2ef267fc3a93361d63c4d6ec521ab311370796fecaeacc9a3c3ff\": container with ID starting with d38a6b1a82c2ef267fc3a93361d63c4d6ec521ab311370796fecaeacc9a3c3ff not found: ID does not exist"
Aug 04 18:15:42.586535 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.586501    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"]
Aug 04 18:15:42.591280 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.591256    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f03bc8-predictor-5cbcf89c6c-chszs"]
Aug 04 18:15:42.871075 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.871043    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"]
Aug 04 18:15:42.871440 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.871427    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a" containerName="storage-initializer"
Aug 04 18:15:42.871489 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.871441    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a" containerName="storage-initializer"
Aug 04 18:15:42.871524 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.871510    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a" containerName="storage-initializer"
Aug 04 18:15:42.871573 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.871564    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a" containerName="storage-initializer"
Aug 04 18:15:42.871608 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.871573    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a" containerName="storage-initializer"
Aug 04 18:15:42.871648 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.871640    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a" containerName="storage-initializer"
Aug 04 18:15:42.879583 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.879560    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:42.882048 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.882026    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Aug 04 18:15:42.882101 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.882080    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Aug 04 18:15:42.886912 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:42.886888    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"]
Aug 04 18:15:43.027334 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.027300    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbhgx\" (UniqueName: \"kubernetes.io/projected/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kube-api-access-mbhgx\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z2hwb\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.027334 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.027340    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z2hwb\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.027627 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.027424    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/311d9be3-117c-4b70-a40a-ae26bf06ac9b-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z2hwb\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.027627 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.027528    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311d9be3-117c-4b70-a40a-ae26bf06ac9b-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z2hwb\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.027627 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.027594    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z2hwb\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.128468 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.128346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z2hwb\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.128468 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.128448    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbhgx\" (UniqueName: \"kubernetes.io/projected/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kube-api-access-mbhgx\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z2hwb\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.128687 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.128485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z2hwb\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.128687 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.128538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/311d9be3-117c-4b70-a40a-ae26bf06ac9b-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z2hwb\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.128687 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.128619    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311d9be3-117c-4b70-a40a-ae26bf06ac9b-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z2hwb\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.128859 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:15:43.128761    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-serving-cert: secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Aug 04 18:15:43.128859 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.128780    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z2hwb\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.128859 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:15:43.128831    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/311d9be3-117c-4b70-a40a-ae26bf06ac9b-proxy-tls podName:311d9be3-117c-4b70-a40a-ae26bf06ac9b nodeName:}" failed. No retries permitted until 2026-08-04 18:15:43.628815927 +0000 UTC m=+942.919430076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/311d9be3-117c-4b70-a40a-ae26bf06ac9b-proxy-tls") pod "isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" (UID: "311d9be3-117c-4b70-a40a-ae26bf06ac9b") : secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Aug 04 18:15:43.129229 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.129209    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311d9be3-117c-4b70-a40a-ae26bf06ac9b-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z2hwb\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.129302 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.129231    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z2hwb\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.138461 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.138433    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbhgx\" (UniqueName: \"kubernetes.io/projected/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kube-api-access-mbhgx\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z2hwb\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.194433 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.194396    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a" path="/var/lib/kubelet/pods/9564eb1f-1ef8-48ac-a6b3-a804d1b01c2a/volumes"
Aug 04 18:15:43.631744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.631714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/311d9be3-117c-4b70-a40a-ae26bf06ac9b-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z2hwb\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.634184 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.634160    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/311d9be3-117c-4b70-a40a-ae26bf06ac9b-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z2hwb\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.791965 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.791928    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:15:43.928230 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:43.928207    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"]
Aug 04 18:15:43.930306 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:15:43.930272    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311d9be3_117c_4b70_a40a_ae26bf06ac9b.slice/crio-70d2f7af1c2ff12f3a1f8007f96103ddc2ef4a60e53c1b99aa96483cc5c283ad WatchSource:0}: Error finding container 70d2f7af1c2ff12f3a1f8007f96103ddc2ef4a60e53c1b99aa96483cc5c283ad: Status 404 returned error can't find the container with id 70d2f7af1c2ff12f3a1f8007f96103ddc2ef4a60e53c1b99aa96483cc5c283ad
Aug 04 18:15:44.557162 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:44.557123    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" event={"ID":"311d9be3-117c-4b70-a40a-ae26bf06ac9b","Type":"ContainerStarted","Data":"c072d5eaf2722e3f06aeb9b8c9f423af4e5c28a95807f9f37711c1bde4f934b4"}
Aug 04 18:15:44.557385 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:44.557235    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" event={"ID":"311d9be3-117c-4b70-a40a-ae26bf06ac9b","Type":"ContainerStarted","Data":"70d2f7af1c2ff12f3a1f8007f96103ddc2ef4a60e53c1b99aa96483cc5c283ad"}
Aug 04 18:15:48.578372 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:48.578320    2566 generic.go:356] "Generic (PLEG): container finished" podID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerID="c072d5eaf2722e3f06aeb9b8c9f423af4e5c28a95807f9f37711c1bde4f934b4" exitCode=0
Aug 04 18:15:48.578736 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:48.578397    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" event={"ID":"311d9be3-117c-4b70-a40a-ae26bf06ac9b","Type":"ContainerDied","Data":"c072d5eaf2722e3f06aeb9b8c9f423af4e5c28a95807f9f37711c1bde4f934b4"}
Aug 04 18:15:52.345338 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:52.344798    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 04 18:15:52.346004 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:15:52.345942    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:16:02.345802 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:02.345752    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 04 18:16:02.346515 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:02.346482    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:16:11.734517 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:11.734480    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" event={"ID":"311d9be3-117c-4b70-a40a-ae26bf06ac9b","Type":"ContainerStarted","Data":"2bfc563040b2cdda1f25336fba434d59ef5e289b2ef54022cd213b72a20bd710"}
Aug 04 18:16:11.734938 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:11.734547    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" event={"ID":"311d9be3-117c-4b70-a40a-ae26bf06ac9b","Type":"ContainerStarted","Data":"e29b2d45e544321e4eff9fbbccafaf6b508cb1e1393d9716f76d6497d41e778e"}
Aug 04 18:16:11.734938 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:11.734750    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:16:11.755997 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:11.755943    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" podStartSLOduration=3.994692641 podStartE2EDuration="29.755926132s" totalImagesPullingTime="22.187765309s" totalInitContainerRuntime="3.573468182s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:16:11.753978102 +0000 UTC m=+971.044592274" watchObservedRunningTime="2026-08-04 18:16:11.755926132 +0000 UTC m=+971.046540303"
Aug 04 18:16:12.345711 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:12.345661    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 04 18:16:12.346156 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:12.346130    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:16:12.739496 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:12.739415    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:16:12.740687 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:12.740657    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Aug 04 18:16:13.745232 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:13.745185    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Aug 04 18:16:18.750416 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:18.750378    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:16:18.750999 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:18.750968    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Aug 04 18:16:21.192285 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:21.192227    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 04 18:16:21.192832 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:21.192703    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:16:28.751512 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:28.751421    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Aug 04 18:16:31.200253 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:31.200226    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:16:31.200783 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:31.200302    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:16:38.750863 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:38.750823    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Aug 04 18:16:44.348198 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.348166    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-qxm5z_99d23590-fd22-4658-a22c-0b74447c2a55/kserve-container/0.log"
Aug 04 18:16:44.556494 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.556462    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"]
Aug 04 18:16:44.556742 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.556711    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container" containerID="cri-o://3e73c2c1453c31a5afa10fe26f9c97615024bf2bbeb81f44d2ea66851dc93f25" gracePeriod=30
Aug 04 18:16:44.556888 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.556729    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent" containerID="cri-o://456233c7649cc5a06baf41c4b5f232d33bd18711f4c1b536d928e0a41428e125" gracePeriod=30
Aug 04 18:16:44.556956 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.556754    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kube-rbac-proxy" containerID="cri-o://e50d2dd1eb203857ca6daf855fe07f964d9ff6baadf75d2758536a33deaa07bf" gracePeriod=30
Aug 04 18:16:44.626001 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.624296    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"]
Aug 04 18:16:44.626001 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.624592    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z" podUID="99d23590-fd22-4658-a22c-0b74447c2a55" containerName="kserve-container" containerID="cri-o://bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d" gracePeriod=30
Aug 04 18:16:44.626001 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.625533    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z" podUID="99d23590-fd22-4658-a22c-0b74447c2a55" containerName="kube-rbac-proxy" containerID="cri-o://16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d" gracePeriod=30
Aug 04 18:16:44.646394 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.646370    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"]
Aug 04 18:16:44.649566 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.649536    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:44.651795 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.651765    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Aug 04 18:16:44.651893 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.651813    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Aug 04 18:16:44.662754 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.662724    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"]
Aug 04 18:16:44.722376 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.722317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-8h78k\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:44.722376 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.722381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzdsr\" (UniqueName: \"kubernetes.io/projected/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kube-api-access-qzdsr\") pod \"isvc-autogluon-v1-predictor-574b5b7495-8h78k\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:44.722584 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.722447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-8h78k\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:44.722584 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.722547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-8h78k\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:44.722584 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.722574    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-8h78k\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:44.823704 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.823669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-8h78k\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:44.823868 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.823731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-8h78k\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:44.823868 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.823762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzdsr\" (UniqueName: \"kubernetes.io/projected/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kube-api-access-qzdsr\") pod \"isvc-autogluon-v1-predictor-574b5b7495-8h78k\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:44.823868 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.823807    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-8h78k\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:44.824031 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.823899    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-8h78k\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:44.824031 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:16:44.823988    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Aug 04 18:16:44.824130 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:16:44.824061    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-proxy-tls podName:a8c7b0ea-6691-4b63-8e35-13a7aa607f6e nodeName:}" failed. No retries permitted until 2026-08-04 18:16:45.324043796 +0000 UTC m=+1004.614657946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-8h78k" (UID: "a8c7b0ea-6691-4b63-8e35-13a7aa607f6e") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Aug 04 18:16:44.824262 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.824242    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-8h78k\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:44.824476 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.824447    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-8h78k\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:44.824638 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.824620    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-8h78k\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:44.833135 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.833112    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzdsr\" (UniqueName: \"kubernetes.io/projected/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kube-api-access-qzdsr\") pod \"isvc-autogluon-v1-predictor-574b5b7495-8h78k\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:44.871089 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.871065    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:16:44.906920 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.906831    2566 generic.go:356] "Generic (PLEG): container finished" podID="99d23590-fd22-4658-a22c-0b74447c2a55" containerID="16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d" exitCode=2
Aug 04 18:16:44.906920 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.906852    2566 generic.go:356] "Generic (PLEG): container finished" podID="99d23590-fd22-4658-a22c-0b74447c2a55" containerID="bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d" exitCode=2
Aug 04 18:16:44.906920 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.906885    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"
Aug 04 18:16:44.906920 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.906908    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z" event={"ID":"99d23590-fd22-4658-a22c-0b74447c2a55","Type":"ContainerDied","Data":"16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d"}
Aug 04 18:16:44.907266 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.906974    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z" event={"ID":"99d23590-fd22-4658-a22c-0b74447c2a55","Type":"ContainerDied","Data":"bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d"}
Aug 04 18:16:44.907266 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.907007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z" event={"ID":"99d23590-fd22-4658-a22c-0b74447c2a55","Type":"ContainerDied","Data":"155a847fe6cf6a3dda09a74e4ad67e97643d2ed9747bc14c02a622714f73c48c"}
Aug 04 18:16:44.907266 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.907039    2566 scope.go:117] "RemoveContainer" containerID="16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d"
Aug 04 18:16:44.909591 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.909570    2566 generic.go:356] "Generic (PLEG): container finished" podID="18e64bad-f6e8-4461-9241-84864c191227" containerID="e50d2dd1eb203857ca6daf855fe07f964d9ff6baadf75d2758536a33deaa07bf" exitCode=2
Aug 04 18:16:44.909713 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.909633    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" event={"ID":"18e64bad-f6e8-4461-9241-84864c191227","Type":"ContainerDied","Data":"e50d2dd1eb203857ca6daf855fe07f964d9ff6baadf75d2758536a33deaa07bf"}
Aug 04 18:16:44.915999 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.915981    2566 scope.go:117] "RemoveContainer" containerID="bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d"
Aug 04 18:16:44.923411 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.923396    2566 scope.go:117] "RemoveContainer" containerID="16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d"
Aug 04 18:16:44.923652 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:16:44.923632    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d\": container with ID starting with 16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d not found: ID does not exist" containerID="16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d"
Aug 04 18:16:44.923698 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.923660    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d"} err="failed to get container status \"16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d\": rpc error: code = NotFound desc = could not find container \"16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d\": container with ID starting with 16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d not found: ID does not exist"
Aug 04 18:16:44.923698 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.923676    2566 scope.go:117] "RemoveContainer" containerID="bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d"
Aug 04 18:16:44.923922 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:16:44.923903    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d\": container with ID starting with bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d not found: ID does not exist" containerID="bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d"
Aug 04 18:16:44.923964 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.923937    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d"} err="failed to get container status \"bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d\": rpc error: code = NotFound desc = could not find container \"bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d\": container with ID starting with bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d not found: ID does not exist"
Aug 04 18:16:44.923964 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.923957    2566 scope.go:117] "RemoveContainer" containerID="16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d"
Aug 04 18:16:44.924183 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.924163    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d"} err="failed to get container status \"16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d\": rpc error: code = NotFound desc = could not find container \"16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d\": container with ID starting with 16cd98d59c8ae3769e640d005a16424918a7896154c663ddc5b15f3f36bb8b9d not found: ID does not exist"
Aug 04 18:16:44.924230 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.924184    2566 scope.go:117] "RemoveContainer" containerID="bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d"
Aug 04 18:16:44.924279 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.924225    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d23590-fd22-4658-a22c-0b74447c2a55-proxy-tls\") pod \"99d23590-fd22-4658-a22c-0b74447c2a55\" (UID: \"99d23590-fd22-4658-a22c-0b74447c2a55\") "
Aug 04 18:16:44.924279 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.924267    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9nkx\" (UniqueName: \"kubernetes.io/projected/99d23590-fd22-4658-a22c-0b74447c2a55-kube-api-access-q9nkx\") pod \"99d23590-fd22-4658-a22c-0b74447c2a55\" (UID: \"99d23590-fd22-4658-a22c-0b74447c2a55\") "
Aug 04 18:16:44.924456 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.924327    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99d23590-fd22-4658-a22c-0b74447c2a55-kube-rbac-proxy-sar-config\") pod \"99d23590-fd22-4658-a22c-0b74447c2a55\" (UID: \"99d23590-fd22-4658-a22c-0b74447c2a55\") "
Aug 04 18:16:44.924552 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.924523    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d"} err="failed to get container status \"bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d\": rpc error: code = NotFound desc = could not find container \"bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d\": container with ID starting with bfa13ba7272e384c6cc0c4903195f2e78145ac127795d161df72f17dc12ff42d not found: ID does not exist"
Aug 04 18:16:44.924766 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.924745    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d23590-fd22-4658-a22c-0b74447c2a55-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "99d23590-fd22-4658-a22c-0b74447c2a55" (UID: "99d23590-fd22-4658-a22c-0b74447c2a55"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:16:44.926265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.926246    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d23590-fd22-4658-a22c-0b74447c2a55-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99d23590-fd22-4658-a22c-0b74447c2a55" (UID: "99d23590-fd22-4658-a22c-0b74447c2a55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:16:44.926343 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:44.926310    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d23590-fd22-4658-a22c-0b74447c2a55-kube-api-access-q9nkx" (OuterVolumeSpecName: "kube-api-access-q9nkx") pod "99d23590-fd22-4658-a22c-0b74447c2a55" (UID: "99d23590-fd22-4658-a22c-0b74447c2a55"). InnerVolumeSpecName "kube-api-access-q9nkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:16:45.025898 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:45.025866    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99d23590-fd22-4658-a22c-0b74447c2a55-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:16:45.025898 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:45.025893    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d23590-fd22-4658-a22c-0b74447c2a55-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:16:45.025898 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:45.025903    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9nkx\" (UniqueName: \"kubernetes.io/projected/99d23590-fd22-4658-a22c-0b74447c2a55-kube-api-access-q9nkx\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:16:45.227616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:45.227584    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"]
Aug 04 18:16:45.230381 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:45.230337    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qxm5z"]
Aug 04 18:16:45.329278 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:45.329244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-8h78k\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:45.331655 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:45.331636    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-8h78k\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:45.601178 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:45.601142    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:16:45.742961 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:45.742931    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"]
Aug 04 18:16:45.918446 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:45.918324    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k" event={"ID":"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e","Type":"ContainerStarted","Data":"962332098c3af1a912914981ae484e4fb22dd25e37af4ac334c0e59ec738dc2f"}
Aug 04 18:16:45.918446 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:45.918407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k" event={"ID":"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e","Type":"ContainerStarted","Data":"531cb5c5ecacf75062e9a1c03f7c8e28883e713344334e009af84f5d4f614cd4"}
Aug 04 18:16:47.194077 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:47.194043    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99d23590-fd22-4658-a22c-0b74447c2a55" path="/var/lib/kubelet/pods/99d23590-fd22-4658-a22c-0b74447c2a55/volumes"
Aug 04 18:16:47.340205 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:47.340153    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Aug 04 18:16:48.751442 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:48.751401    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Aug 04 18:16:49.940702 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:49.940670    2566 generic.go:356] "Generic (PLEG): container finished" podID="18e64bad-f6e8-4461-9241-84864c191227" containerID="3e73c2c1453c31a5afa10fe26f9c97615024bf2bbeb81f44d2ea66851dc93f25" exitCode=0
Aug 04 18:16:49.941160 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:49.940747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" event={"ID":"18e64bad-f6e8-4461-9241-84864c191227","Type":"ContainerDied","Data":"3e73c2c1453c31a5afa10fe26f9c97615024bf2bbeb81f44d2ea66851dc93f25"}
Aug 04 18:16:51.192204 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:51.192151    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 04 18:16:51.192686 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:51.192449    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:16:52.340079 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:52.340028    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Aug 04 18:16:57.340793 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:57.340747    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Aug 04 18:16:57.341180 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:57.340875    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:16:58.751766 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:16:58.751730    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Aug 04 18:17:01.192175 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:01.192130    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 04 18:17:01.192682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:01.192482    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:17:02.340822 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:02.340768    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Aug 04 18:17:06.020046 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:06.020016    2566 generic.go:356] "Generic (PLEG): container finished" podID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" containerID="962332098c3af1a912914981ae484e4fb22dd25e37af4ac334c0e59ec738dc2f" exitCode=0
Aug 04 18:17:06.020439 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:06.020088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k" event={"ID":"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e","Type":"ContainerDied","Data":"962332098c3af1a912914981ae484e4fb22dd25e37af4ac334c0e59ec738dc2f"}
Aug 04 18:17:07.340764 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:07.340718    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Aug 04 18:17:08.751848 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:08.751800    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Aug 04 18:17:11.192219 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:11.192155    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Aug 04 18:17:11.192909 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:11.192793    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:17:11.196783 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:11.196746    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:17:11.196980 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:11.196965    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:17:12.340538 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:12.340492    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Aug 04 18:17:14.735440 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:14.735414    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:17:14.904871 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:14.904771    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18e64bad-f6e8-4461-9241-84864c191227-proxy-tls\") pod \"18e64bad-f6e8-4461-9241-84864c191227\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") "
Aug 04 18:17:14.905055 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:14.904877    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bqrk\" (UniqueName: \"kubernetes.io/projected/18e64bad-f6e8-4461-9241-84864c191227-kube-api-access-4bqrk\") pod \"18e64bad-f6e8-4461-9241-84864c191227\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") "
Aug 04 18:17:14.905055 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:14.904922    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18e64bad-f6e8-4461-9241-84864c191227-kube-rbac-proxy-sar-config\") pod \"18e64bad-f6e8-4461-9241-84864c191227\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") "
Aug 04 18:17:14.905055 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:14.904954    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18e64bad-f6e8-4461-9241-84864c191227-kserve-provision-location\") pod \"18e64bad-f6e8-4461-9241-84864c191227\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") "
Aug 04 18:17:14.905055 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:14.904976    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18e64bad-f6e8-4461-9241-84864c191227-cabundle-cert\") pod \"18e64bad-f6e8-4461-9241-84864c191227\" (UID: \"18e64bad-f6e8-4461-9241-84864c191227\") "
Aug 04 18:17:14.905304 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:14.905269    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e64bad-f6e8-4461-9241-84864c191227-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "18e64bad-f6e8-4461-9241-84864c191227" (UID: "18e64bad-f6e8-4461-9241-84864c191227"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:17:14.905460 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:14.905305    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e64bad-f6e8-4461-9241-84864c191227-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "18e64bad-f6e8-4461-9241-84864c191227" (UID: "18e64bad-f6e8-4461-9241-84864c191227"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:17:14.905460 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:14.905416    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e64bad-f6e8-4461-9241-84864c191227-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "18e64bad-f6e8-4461-9241-84864c191227" (UID: "18e64bad-f6e8-4461-9241-84864c191227"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:17:14.907051 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:14.907020    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e64bad-f6e8-4461-9241-84864c191227-kube-api-access-4bqrk" (OuterVolumeSpecName: "kube-api-access-4bqrk") pod "18e64bad-f6e8-4461-9241-84864c191227" (UID: "18e64bad-f6e8-4461-9241-84864c191227"). InnerVolumeSpecName "kube-api-access-4bqrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:17:14.907150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:14.907066    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e64bad-f6e8-4461-9241-84864c191227-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "18e64bad-f6e8-4461-9241-84864c191227" (UID: "18e64bad-f6e8-4461-9241-84864c191227"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:17:15.006424 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.006376    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18e64bad-f6e8-4461-9241-84864c191227-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:17:15.006424 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.006417    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bqrk\" (UniqueName: \"kubernetes.io/projected/18e64bad-f6e8-4461-9241-84864c191227-kube-api-access-4bqrk\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:17:15.006424 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.006428    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18e64bad-f6e8-4461-9241-84864c191227-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:17:15.006424 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.006437    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18e64bad-f6e8-4461-9241-84864c191227-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:17:15.006424 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.006446    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18e64bad-f6e8-4461-9241-84864c191227-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:17:15.081212 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.081180    2566 generic.go:356] "Generic (PLEG): container finished" podID="18e64bad-f6e8-4461-9241-84864c191227" containerID="456233c7649cc5a06baf41c4b5f232d33bd18711f4c1b536d928e0a41428e125" exitCode=0
Aug 04 18:17:15.081426 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.081244    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" event={"ID":"18e64bad-f6e8-4461-9241-84864c191227","Type":"ContainerDied","Data":"456233c7649cc5a06baf41c4b5f232d33bd18711f4c1b536d928e0a41428e125"}
Aug 04 18:17:15.081426 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.081274    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"
Aug 04 18:17:15.081426 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.081339    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4" event={"ID":"18e64bad-f6e8-4461-9241-84864c191227","Type":"ContainerDied","Data":"2fbf0db4eb4ea6280937e4d24fb3036d1eb7e6f12605cff668aa73c901fd2cc6"}
Aug 04 18:17:15.081426 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.081386    2566 scope.go:117] "RemoveContainer" containerID="456233c7649cc5a06baf41c4b5f232d33bd18711f4c1b536d928e0a41428e125"
Aug 04 18:17:15.090813 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.090796    2566 scope.go:117] "RemoveContainer" containerID="e50d2dd1eb203857ca6daf855fe07f964d9ff6baadf75d2758536a33deaa07bf"
Aug 04 18:17:15.098257 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.098242    2566 scope.go:117] "RemoveContainer" containerID="3e73c2c1453c31a5afa10fe26f9c97615024bf2bbeb81f44d2ea66851dc93f25"
Aug 04 18:17:15.105943 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.105918    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"]
Aug 04 18:17:15.106271 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.106253    2566 scope.go:117] "RemoveContainer" containerID="68607f9ee7cf40f2663da7bf76b4c96092aa87a80d3a40005c6555b339de0834"
Aug 04 18:17:15.110965 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.110939    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6cb98666b8-2xpg4"]
Aug 04 18:17:15.114641 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.114623    2566 scope.go:117] "RemoveContainer" containerID="456233c7649cc5a06baf41c4b5f232d33bd18711f4c1b536d928e0a41428e125"
Aug 04 18:17:15.114927 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:17:15.114908    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456233c7649cc5a06baf41c4b5f232d33bd18711f4c1b536d928e0a41428e125\": container with ID starting with 456233c7649cc5a06baf41c4b5f232d33bd18711f4c1b536d928e0a41428e125 not found: ID does not exist" containerID="456233c7649cc5a06baf41c4b5f232d33bd18711f4c1b536d928e0a41428e125"
Aug 04 18:17:15.114985 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.114936    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456233c7649cc5a06baf41c4b5f232d33bd18711f4c1b536d928e0a41428e125"} err="failed to get container status \"456233c7649cc5a06baf41c4b5f232d33bd18711f4c1b536d928e0a41428e125\": rpc error: code = NotFound desc = could not find container \"456233c7649cc5a06baf41c4b5f232d33bd18711f4c1b536d928e0a41428e125\": container with ID starting with 456233c7649cc5a06baf41c4b5f232d33bd18711f4c1b536d928e0a41428e125 not found: ID does not exist"
Aug 04 18:17:15.114985 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.114955    2566 scope.go:117] "RemoveContainer" containerID="e50d2dd1eb203857ca6daf855fe07f964d9ff6baadf75d2758536a33deaa07bf"
Aug 04 18:17:15.115183 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:17:15.115164    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50d2dd1eb203857ca6daf855fe07f964d9ff6baadf75d2758536a33deaa07bf\": container with ID starting with e50d2dd1eb203857ca6daf855fe07f964d9ff6baadf75d2758536a33deaa07bf not found: ID does not exist" containerID="e50d2dd1eb203857ca6daf855fe07f964d9ff6baadf75d2758536a33deaa07bf"
Aug 04 18:17:15.115256 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.115193    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50d2dd1eb203857ca6daf855fe07f964d9ff6baadf75d2758536a33deaa07bf"} err="failed to get container status \"e50d2dd1eb203857ca6daf855fe07f964d9ff6baadf75d2758536a33deaa07bf\": rpc error: code = NotFound desc = could not find container \"e50d2dd1eb203857ca6daf855fe07f964d9ff6baadf75d2758536a33deaa07bf\": container with ID starting with e50d2dd1eb203857ca6daf855fe07f964d9ff6baadf75d2758536a33deaa07bf not found: ID does not exist"
Aug 04 18:17:15.115256 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.115216    2566 scope.go:117] "RemoveContainer" containerID="3e73c2c1453c31a5afa10fe26f9c97615024bf2bbeb81f44d2ea66851dc93f25"
Aug 04 18:17:15.115467 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:17:15.115443    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e73c2c1453c31a5afa10fe26f9c97615024bf2bbeb81f44d2ea66851dc93f25\": container with ID starting with 3e73c2c1453c31a5afa10fe26f9c97615024bf2bbeb81f44d2ea66851dc93f25 not found: ID does not exist" containerID="3e73c2c1453c31a5afa10fe26f9c97615024bf2bbeb81f44d2ea66851dc93f25"
Aug 04 18:17:15.115550 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.115466    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e73c2c1453c31a5afa10fe26f9c97615024bf2bbeb81f44d2ea66851dc93f25"} err="failed to get container status \"3e73c2c1453c31a5afa10fe26f9c97615024bf2bbeb81f44d2ea66851dc93f25\": rpc error: code = NotFound desc = could not find container \"3e73c2c1453c31a5afa10fe26f9c97615024bf2bbeb81f44d2ea66851dc93f25\": container with ID starting with 3e73c2c1453c31a5afa10fe26f9c97615024bf2bbeb81f44d2ea66851dc93f25 not found: ID does not exist"
Aug 04 18:17:15.115550 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.115485    2566 scope.go:117] "RemoveContainer" containerID="68607f9ee7cf40f2663da7bf76b4c96092aa87a80d3a40005c6555b339de0834"
Aug 04 18:17:15.115684 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:17:15.115669    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68607f9ee7cf40f2663da7bf76b4c96092aa87a80d3a40005c6555b339de0834\": container with ID starting with 68607f9ee7cf40f2663da7bf76b4c96092aa87a80d3a40005c6555b339de0834 not found: ID does not exist" containerID="68607f9ee7cf40f2663da7bf76b4c96092aa87a80d3a40005c6555b339de0834"
Aug 04 18:17:15.115721 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.115687    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68607f9ee7cf40f2663da7bf76b4c96092aa87a80d3a40005c6555b339de0834"} err="failed to get container status \"68607f9ee7cf40f2663da7bf76b4c96092aa87a80d3a40005c6555b339de0834\": rpc error: code = NotFound desc = could not find container \"68607f9ee7cf40f2663da7bf76b4c96092aa87a80d3a40005c6555b339de0834\": container with ID starting with 68607f9ee7cf40f2663da7bf76b4c96092aa87a80d3a40005c6555b339de0834 not found: ID does not exist"
Aug 04 18:17:15.195703 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:15.195632    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18e64bad-f6e8-4461-9241-84864c191227" path="/var/lib/kubelet/pods/18e64bad-f6e8-4461-9241-84864c191227/volumes"
Aug 04 18:17:18.751328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:18.751285    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Aug 04 18:17:28.752278 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:28.752242    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:17:32.961951 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:32.961915    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"]
Aug 04 18:17:32.962475 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:32.962254    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kserve-container" containerID="cri-o://e29b2d45e544321e4eff9fbbccafaf6b508cb1e1393d9716f76d6497d41e778e" gracePeriod=30
Aug 04 18:17:32.962475 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:32.962323    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kube-rbac-proxy" containerID="cri-o://2bfc563040b2cdda1f25336fba434d59ef5e289b2ef54022cd213b72a20bd710" gracePeriod=30
Aug 04 18:17:33.183145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:33.183114    2566 generic.go:356] "Generic (PLEG): container finished" podID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerID="2bfc563040b2cdda1f25336fba434d59ef5e289b2ef54022cd213b72a20bd710" exitCode=2
Aug 04 18:17:33.183301 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:33.183184    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" event={"ID":"311d9be3-117c-4b70-a40a-ae26bf06ac9b","Type":"ContainerDied","Data":"2bfc563040b2cdda1f25336fba434d59ef5e289b2ef54022cd213b72a20bd710"}
Aug 04 18:17:33.746105 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:33.746061    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Aug 04 18:17:38.169093 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.169065    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:17:38.211304 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211204    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"]
Aug 04 18:17:38.211777 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211750    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d23590-fd22-4658-a22c-0b74447c2a55" containerName="kube-rbac-proxy"
Aug 04 18:17:38.211777 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211774    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d23590-fd22-4658-a22c-0b74447c2a55" containerName="kube-rbac-proxy"
Aug 04 18:17:38.211971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211792    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kserve-container"
Aug 04 18:17:38.211971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211800    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kserve-container"
Aug 04 18:17:38.211971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211817    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d23590-fd22-4658-a22c-0b74447c2a55" containerName="kserve-container"
Aug 04 18:17:38.211971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211825    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d23590-fd22-4658-a22c-0b74447c2a55" containerName="kserve-container"
Aug 04 18:17:38.211971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211835    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="storage-initializer"
Aug 04 18:17:38.211971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211844    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="storage-initializer"
Aug 04 18:17:38.211971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211853    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container"
Aug 04 18:17:38.211971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211862    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container"
Aug 04 18:17:38.211971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211869    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent"
Aug 04 18:17:38.211971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211874    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent"
Aug 04 18:17:38.211971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211890    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kube-rbac-proxy"
Aug 04 18:17:38.211971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211895    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kube-rbac-proxy"
Aug 04 18:17:38.211971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211909    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kube-rbac-proxy"
Aug 04 18:17:38.211971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211917    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kube-rbac-proxy"
Aug 04 18:17:38.211971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211928    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="storage-initializer"
Aug 04 18:17:38.211971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211934    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="storage-initializer"
Aug 04 18:17:38.212616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.211991    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="99d23590-fd22-4658-a22c-0b74447c2a55" containerName="kube-rbac-proxy"
Aug 04 18:17:38.212616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.212000    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="99d23590-fd22-4658-a22c-0b74447c2a55" containerName="kserve-container"
Aug 04 18:17:38.212616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.212007    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kube-rbac-proxy"
Aug 04 18:17:38.212616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.212014    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kube-rbac-proxy"
Aug 04 18:17:38.212616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.212020    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="agent"
Aug 04 18:17:38.212616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.212027    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerName="kserve-container"
Aug 04 18:17:38.212616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.212033    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="18e64bad-f6e8-4461-9241-84864c191227" containerName="kserve-container"
Aug 04 18:17:38.213300 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.213277    2566 generic.go:356] "Generic (PLEG): container finished" podID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" containerID="e29b2d45e544321e4eff9fbbccafaf6b508cb1e1393d9716f76d6497d41e778e" exitCode=0
Aug 04 18:17:38.213414 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.213395    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"
Aug 04 18:17:38.221415 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.221390    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/311d9be3-117c-4b70-a40a-ae26bf06ac9b-proxy-tls\") pod \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") "
Aug 04 18:17:38.221522 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.221426    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311d9be3-117c-4b70-a40a-ae26bf06ac9b-cabundle-cert\") pod \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") "
Aug 04 18:17:38.221522 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.221470    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kserve-provision-location\") pod \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") "
Aug 04 18:17:38.221629 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.221532    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbhgx\" (UniqueName: \"kubernetes.io/projected/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kube-api-access-mbhgx\") pod \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") "
Aug 04 18:17:38.221629 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.221608    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kube-rbac-proxy-sar-config\") pod \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\" (UID: \"311d9be3-117c-4b70-a40a-ae26bf06ac9b\") "
Aug 04 18:17:38.221944 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.221842    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311d9be3-117c-4b70-a40a-ae26bf06ac9b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "311d9be3-117c-4b70-a40a-ae26bf06ac9b" (UID: "311d9be3-117c-4b70-a40a-ae26bf06ac9b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:17:38.221944 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.221899    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "311d9be3-117c-4b70-a40a-ae26bf06ac9b" (UID: "311d9be3-117c-4b70-a40a-ae26bf06ac9b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:17:38.222136 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.222102    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "311d9be3-117c-4b70-a40a-ae26bf06ac9b" (UID: "311d9be3-117c-4b70-a40a-ae26bf06ac9b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:17:38.224701 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.224670    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311d9be3-117c-4b70-a40a-ae26bf06ac9b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "311d9be3-117c-4b70-a40a-ae26bf06ac9b" (UID: "311d9be3-117c-4b70-a40a-ae26bf06ac9b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:17:38.224797 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.224734    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kube-api-access-mbhgx" (OuterVolumeSpecName: "kube-api-access-mbhgx") pod "311d9be3-117c-4b70-a40a-ae26bf06ac9b" (UID: "311d9be3-117c-4b70-a40a-ae26bf06ac9b"). InnerVolumeSpecName "kube-api-access-mbhgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:17:38.226908 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.226885    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" event={"ID":"311d9be3-117c-4b70-a40a-ae26bf06ac9b","Type":"ContainerDied","Data":"e29b2d45e544321e4eff9fbbccafaf6b508cb1e1393d9716f76d6497d41e778e"}
Aug 04 18:17:38.227017 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.226937    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb" event={"ID":"311d9be3-117c-4b70-a40a-ae26bf06ac9b","Type":"ContainerDied","Data":"70d2f7af1c2ff12f3a1f8007f96103ddc2ef4a60e53c1b99aa96483cc5c283ad"}
Aug 04 18:17:38.227017 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.226976    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"]
Aug 04 18:17:38.227017 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.227005    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.227153 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.227022    2566 scope.go:117] "RemoveContainer" containerID="2bfc563040b2cdda1f25336fba434d59ef5e289b2ef54022cd213b72a20bd710"
Aug 04 18:17:38.229477 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.229458    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Aug 04 18:17:38.229576 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.229496    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Aug 04 18:17:38.242616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.242593    2566 scope.go:117] "RemoveContainer" containerID="e29b2d45e544321e4eff9fbbccafaf6b508cb1e1393d9716f76d6497d41e778e"
Aug 04 18:17:38.252266 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.252246    2566 scope.go:117] "RemoveContainer" containerID="c072d5eaf2722e3f06aeb9b8c9f423af4e5c28a95807f9f37711c1bde4f934b4"
Aug 04 18:17:38.259982 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.259963    2566 scope.go:117] "RemoveContainer" containerID="2bfc563040b2cdda1f25336fba434d59ef5e289b2ef54022cd213b72a20bd710"
Aug 04 18:17:38.260240 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:17:38.260219    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bfc563040b2cdda1f25336fba434d59ef5e289b2ef54022cd213b72a20bd710\": container with ID starting with 2bfc563040b2cdda1f25336fba434d59ef5e289b2ef54022cd213b72a20bd710 not found: ID does not exist" containerID="2bfc563040b2cdda1f25336fba434d59ef5e289b2ef54022cd213b72a20bd710"
Aug 04 18:17:38.260322 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.260254    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfc563040b2cdda1f25336fba434d59ef5e289b2ef54022cd213b72a20bd710"} err="failed to get container status \"2bfc563040b2cdda1f25336fba434d59ef5e289b2ef54022cd213b72a20bd710\": rpc error: code = NotFound desc = could not find container \"2bfc563040b2cdda1f25336fba434d59ef5e289b2ef54022cd213b72a20bd710\": container with ID starting with 2bfc563040b2cdda1f25336fba434d59ef5e289b2ef54022cd213b72a20bd710 not found: ID does not exist"
Aug 04 18:17:38.260322 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.260280    2566 scope.go:117] "RemoveContainer" containerID="e29b2d45e544321e4eff9fbbccafaf6b508cb1e1393d9716f76d6497d41e778e"
Aug 04 18:17:38.260586 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:17:38.260569    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e29b2d45e544321e4eff9fbbccafaf6b508cb1e1393d9716f76d6497d41e778e\": container with ID starting with e29b2d45e544321e4eff9fbbccafaf6b508cb1e1393d9716f76d6497d41e778e not found: ID does not exist" containerID="e29b2d45e544321e4eff9fbbccafaf6b508cb1e1393d9716f76d6497d41e778e"
Aug 04 18:17:38.260622 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.260593    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e29b2d45e544321e4eff9fbbccafaf6b508cb1e1393d9716f76d6497d41e778e"} err="failed to get container status \"e29b2d45e544321e4eff9fbbccafaf6b508cb1e1393d9716f76d6497d41e778e\": rpc error: code = NotFound desc = could not find container \"e29b2d45e544321e4eff9fbbccafaf6b508cb1e1393d9716f76d6497d41e778e\": container with ID starting with e29b2d45e544321e4eff9fbbccafaf6b508cb1e1393d9716f76d6497d41e778e not found: ID does not exist"
Aug 04 18:17:38.260622 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.260608    2566 scope.go:117] "RemoveContainer" containerID="c072d5eaf2722e3f06aeb9b8c9f423af4e5c28a95807f9f37711c1bde4f934b4"
Aug 04 18:17:38.260842 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:17:38.260824    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c072d5eaf2722e3f06aeb9b8c9f423af4e5c28a95807f9f37711c1bde4f934b4\": container with ID starting with c072d5eaf2722e3f06aeb9b8c9f423af4e5c28a95807f9f37711c1bde4f934b4 not found: ID does not exist" containerID="c072d5eaf2722e3f06aeb9b8c9f423af4e5c28a95807f9f37711c1bde4f934b4"
Aug 04 18:17:38.260886 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.260848    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c072d5eaf2722e3f06aeb9b8c9f423af4e5c28a95807f9f37711c1bde4f934b4"} err="failed to get container status \"c072d5eaf2722e3f06aeb9b8c9f423af4e5c28a95807f9f37711c1bde4f934b4\": rpc error: code = NotFound desc = could not find container \"c072d5eaf2722e3f06aeb9b8c9f423af4e5c28a95807f9f37711c1bde4f934b4\": container with ID starting with c072d5eaf2722e3f06aeb9b8c9f423af4e5c28a95807f9f37711c1bde4f934b4 not found: ID does not exist"
Aug 04 18:17:38.322555 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.322512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-blrnm\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.322746 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.322601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-blrnm\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.322746 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.322632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-blrnm\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.322746 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.322737    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-blrnm\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.322949 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.322772    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvghk\" (UniqueName: \"kubernetes.io/projected/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kube-api-access-zvghk\") pod \"isvc-predictive-xgboost-predictor-695f978978-blrnm\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.322949 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.322846    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:17:38.322949 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.322861    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/311d9be3-117c-4b70-a40a-ae26bf06ac9b-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:17:38.322949 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.322875    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311d9be3-117c-4b70-a40a-ae26bf06ac9b-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:17:38.322949 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.322888    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:17:38.322949 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.322901    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbhgx\" (UniqueName: \"kubernetes.io/projected/311d9be3-117c-4b70-a40a-ae26bf06ac9b-kube-api-access-mbhgx\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:17:38.423501 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.423462    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-blrnm\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.423685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.423527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-blrnm\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.423685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.423552    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvghk\" (UniqueName: \"kubernetes.io/projected/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kube-api-access-zvghk\") pod \"isvc-predictive-xgboost-predictor-695f978978-blrnm\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.423685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.423612    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-blrnm\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.423685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.423664    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-blrnm\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.424258 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.423961    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-blrnm\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.424258 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.424253    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-blrnm\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.424585 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.424561    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-blrnm\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.426463 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.426444    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-blrnm\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.433692 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.433667    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvghk\" (UniqueName: \"kubernetes.io/projected/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kube-api-access-zvghk\") pod \"isvc-predictive-xgboost-predictor-695f978978-blrnm\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.541584 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.541545    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"]
Aug 04 18:17:38.548837 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.548813    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:38.549223 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.549200    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z2hwb"]
Aug 04 18:17:38.734039 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:38.734006    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"]
Aug 04 18:17:38.736223 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:17:38.736195    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2dfb04f_e78b_439d_8db7_6d9daf9e00de.slice/crio-51d20affc861f671bef9891390a917bc64953f3d5c7042d26ab2d8dc2c9215ca WatchSource:0}: Error finding container 51d20affc861f671bef9891390a917bc64953f3d5c7042d26ab2d8dc2c9215ca: Status 404 returned error can't find the container with id 51d20affc861f671bef9891390a917bc64953f3d5c7042d26ab2d8dc2c9215ca
Aug 04 18:17:39.195737 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:39.195699    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="311d9be3-117c-4b70-a40a-ae26bf06ac9b" path="/var/lib/kubelet/pods/311d9be3-117c-4b70-a40a-ae26bf06ac9b/volumes"
Aug 04 18:17:39.221963 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:39.221931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" event={"ID":"f2dfb04f-e78b-439d-8db7-6d9daf9e00de","Type":"ContainerStarted","Data":"5cba2d9e83bf779531c2e882376b789369397e891a0f9b323511d94d77638def"}
Aug 04 18:17:39.222150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:39.221998    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" event={"ID":"f2dfb04f-e78b-439d-8db7-6d9daf9e00de","Type":"ContainerStarted","Data":"51d20affc861f671bef9891390a917bc64953f3d5c7042d26ab2d8dc2c9215ca"}
Aug 04 18:17:43.244136 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:43.244100    2566 generic.go:356] "Generic (PLEG): container finished" podID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerID="5cba2d9e83bf779531c2e882376b789369397e891a0f9b323511d94d77638def" exitCode=0
Aug 04 18:17:43.244659 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:43.244171    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" event={"ID":"f2dfb04f-e78b-439d-8db7-6d9daf9e00de","Type":"ContainerDied","Data":"5cba2d9e83bf779531c2e882376b789369397e891a0f9b323511d94d77638def"}
Aug 04 18:17:44.253287 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:44.253254    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" event={"ID":"f2dfb04f-e78b-439d-8db7-6d9daf9e00de","Type":"ContainerStarted","Data":"0dd4e2ef096bf3a63128e3c155bf65419860caebc8e81ef457184435ac81577f"}
Aug 04 18:17:44.253693 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:44.253342    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" event={"ID":"f2dfb04f-e78b-439d-8db7-6d9daf9e00de","Type":"ContainerStarted","Data":"17cc7a9a338d8c913349ccc41cce5e6c3876364068cc0cf23df963752f1e089a"}
Aug 04 18:17:44.253693 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:44.253644    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:44.253825 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:44.253806    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:44.255055 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:44.255027    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 04 18:17:44.280677 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:44.280626    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" podStartSLOduration=2.555263313 podStartE2EDuration="6.280608299s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.725344986s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:44.27875755 +0000 UTC m=+1063.569371723" watchObservedRunningTime="2026-08-04 18:17:44.280608299 +0000 UTC m=+1063.571222470"
Aug 04 18:17:45.258769 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:45.258729    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 04 18:17:50.263767 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:50.263738    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:17:50.264507 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:17:50.264469    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 04 18:18:00.264836 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:18:00.264750    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 04 18:18:10.264907 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:18:10.264866    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 04 18:18:20.264940 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:18:20.264895    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 04 18:18:30.265079 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:18:30.265032    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 04 18:18:40.265253 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:18:40.265201    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 04 18:18:50.264569 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:18:50.264526    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 04 18:19:00.264824 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:00.264762    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 04 18:19:10.265044 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:10.265006    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:19:18.330988 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:18.330959    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"]
Aug 04 18:19:18.331448 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:18.331243    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kserve-container" containerID="cri-o://17cc7a9a338d8c913349ccc41cce5e6c3876364068cc0cf23df963752f1e089a" gracePeriod=30
Aug 04 18:19:18.331448 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:18.331375    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kube-rbac-proxy" containerID="cri-o://0dd4e2ef096bf3a63128e3c155bf65419860caebc8e81ef457184435ac81577f" gracePeriod=30
Aug 04 18:19:18.792504 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:18.792415    2566 generic.go:356] "Generic (PLEG): container finished" podID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerID="0dd4e2ef096bf3a63128e3c155bf65419860caebc8e81ef457184435ac81577f" exitCode=2
Aug 04 18:19:18.792504 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:18.792492    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" event={"ID":"f2dfb04f-e78b-439d-8db7-6d9daf9e00de","Type":"ContainerDied","Data":"0dd4e2ef096bf3a63128e3c155bf65419860caebc8e81ef457184435ac81577f"}
Aug 04 18:19:18.794645 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:18.794619    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k" event={"ID":"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e","Type":"ContainerStarted","Data":"f912b0e66a4b1547b55e587f5c752fdb7af21d7491715fe093f20b0dce95a978"}
Aug 04 18:19:18.794776 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:18.794676    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k" event={"ID":"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e","Type":"ContainerStarted","Data":"242ed36158f5202ac70c7ed199d0af6b49fce98501e3e4151d8f56141d760ce2"}
Aug 04 18:19:18.794832 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:18.794774    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:19:18.819009 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:18.818950    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k" podStartSLOduration=2.943972559 podStartE2EDuration="2m34.818934846s" totalImagesPullingTime="2m11.841749163s" totalInitContainerRuntime="20.033213124s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:19:18.818065835 +0000 UTC m=+1158.108680056" watchObservedRunningTime="2026-08-04 18:19:18.818934846 +0000 UTC m=+1158.109549055"
Aug 04 18:19:19.801095 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:19.801058    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:19:19.802225 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:19.802199    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k" podUID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Aug 04 18:19:20.259429 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:20.259312    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Aug 04 18:19:20.265194 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:20.265163    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Aug 04 18:19:20.805762 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:20.805724    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k" podUID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Aug 04 18:19:23.131329 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.131253    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:19:23.231313 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.231271    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-cabundle-cert\") pod \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") "
Aug 04 18:19:23.231538 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.231341    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kube-rbac-proxy-sar-config\") pod \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") "
Aug 04 18:19:23.231538 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.231405    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-proxy-tls\") pod \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") "
Aug 04 18:19:23.231538 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.231475    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvghk\" (UniqueName: \"kubernetes.io/projected/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kube-api-access-zvghk\") pod \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") "
Aug 04 18:19:23.231726 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.231589    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kserve-provision-location\") pod \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\" (UID: \"f2dfb04f-e78b-439d-8db7-6d9daf9e00de\") "
Aug 04 18:19:23.231785 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.231716    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2dfb04f-e78b-439d-8db7-6d9daf9e00de" (UID: "f2dfb04f-e78b-439d-8db7-6d9daf9e00de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:19:23.231785 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.231744    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f2dfb04f-e78b-439d-8db7-6d9daf9e00de" (UID: "f2dfb04f-e78b-439d-8db7-6d9daf9e00de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:19:23.231915 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.231895    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:19:23.231974 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.231917    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:19:23.232029 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.231996    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2dfb04f-e78b-439d-8db7-6d9daf9e00de" (UID: "f2dfb04f-e78b-439d-8db7-6d9daf9e00de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:19:23.233769 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.233743    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2dfb04f-e78b-439d-8db7-6d9daf9e00de" (UID: "f2dfb04f-e78b-439d-8db7-6d9daf9e00de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:19:23.233870 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.233770    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kube-api-access-zvghk" (OuterVolumeSpecName: "kube-api-access-zvghk") pod "f2dfb04f-e78b-439d-8db7-6d9daf9e00de" (UID: "f2dfb04f-e78b-439d-8db7-6d9daf9e00de"). InnerVolumeSpecName "kube-api-access-zvghk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:19:23.333622 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.333587    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:19:23.333622 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.333622    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvghk\" (UniqueName: \"kubernetes.io/projected/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kube-api-access-zvghk\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:19:23.333818 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.333640    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2dfb04f-e78b-439d-8db7-6d9daf9e00de-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:19:23.505688 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.505605    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"]
Aug 04 18:19:23.506201 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.506179    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kserve-container"
Aug 04 18:19:23.506244 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.506208    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kserve-container"
Aug 04 18:19:23.506244 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.506231    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="storage-initializer"
Aug 04 18:19:23.506244 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.506240    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="storage-initializer"
Aug 04 18:19:23.506332 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.506250    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kube-rbac-proxy"
Aug 04 18:19:23.506332 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.506259    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kube-rbac-proxy"
Aug 04 18:19:23.506423 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.506343    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kube-rbac-proxy"
Aug 04 18:19:23.506423 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.506378    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerName="kserve-container"
Aug 04 18:19:23.537539 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.537506    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"]
Aug 04 18:19:23.537691 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.537628    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.540300 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.540277    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Aug 04 18:19:23.540300 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.540295    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Aug 04 18:19:23.736952 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.736908    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/959496f4-2deb-495f-a8b3-0fe4aacc6634-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.736952 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.736952    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/959496f4-2deb-495f-a8b3-0fe4aacc6634-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.737218 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.736985    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmb4n\" (UniqueName: \"kubernetes.io/projected/959496f4-2deb-495f-a8b3-0fe4aacc6634-kube-api-access-rmb4n\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.737218 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.737042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/959496f4-2deb-495f-a8b3-0fe4aacc6634-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.737218 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.737079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/959496f4-2deb-495f-a8b3-0fe4aacc6634-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.824076 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.824045    2566 generic.go:356] "Generic (PLEG): container finished" podID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" containerID="17cc7a9a338d8c913349ccc41cce5e6c3876364068cc0cf23df963752f1e089a" exitCode=0
Aug 04 18:19:23.824286 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.824098    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"
Aug 04 18:19:23.824286 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.824139    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" event={"ID":"f2dfb04f-e78b-439d-8db7-6d9daf9e00de","Type":"ContainerDied","Data":"17cc7a9a338d8c913349ccc41cce5e6c3876364068cc0cf23df963752f1e089a"}
Aug 04 18:19:23.824286 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.824204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm" event={"ID":"f2dfb04f-e78b-439d-8db7-6d9daf9e00de","Type":"ContainerDied","Data":"51d20affc861f671bef9891390a917bc64953f3d5c7042d26ab2d8dc2c9215ca"}
Aug 04 18:19:23.824286 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.824232    2566 scope.go:117] "RemoveContainer" containerID="0dd4e2ef096bf3a63128e3c155bf65419860caebc8e81ef457184435ac81577f"
Aug 04 18:19:23.833905 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.833885    2566 scope.go:117] "RemoveContainer" containerID="17cc7a9a338d8c913349ccc41cce5e6c3876364068cc0cf23df963752f1e089a"
Aug 04 18:19:23.837704 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.837674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/959496f4-2deb-495f-a8b3-0fe4aacc6634-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.837822 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.837741    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/959496f4-2deb-495f-a8b3-0fe4aacc6634-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.837822 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.837785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmb4n\" (UniqueName: \"kubernetes.io/projected/959496f4-2deb-495f-a8b3-0fe4aacc6634-kube-api-access-rmb4n\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.837822 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.837813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/959496f4-2deb-495f-a8b3-0fe4aacc6634-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.837999 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.837847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/959496f4-2deb-495f-a8b3-0fe4aacc6634-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.838119 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.838099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/959496f4-2deb-495f-a8b3-0fe4aacc6634-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.838661 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.838638    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/959496f4-2deb-495f-a8b3-0fe4aacc6634-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.838745 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.838656    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/959496f4-2deb-495f-a8b3-0fe4aacc6634-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.840719 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.840698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/959496f4-2deb-495f-a8b3-0fe4aacc6634-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.843397 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.843369    2566 scope.go:117] "RemoveContainer" containerID="5cba2d9e83bf779531c2e882376b789369397e891a0f9b323511d94d77638def"
Aug 04 18:19:23.847454 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.847431    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmb4n\" (UniqueName: \"kubernetes.io/projected/959496f4-2deb-495f-a8b3-0fe4aacc6634-kube-api-access-rmb4n\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.848220 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.848103    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:23.850279 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.850255    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"]
Aug 04 18:19:23.853594 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.853299    2566 scope.go:117] "RemoveContainer" containerID="0dd4e2ef096bf3a63128e3c155bf65419860caebc8e81ef457184435ac81577f"
Aug 04 18:19:23.859939 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:19:23.859913    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd4e2ef096bf3a63128e3c155bf65419860caebc8e81ef457184435ac81577f\": container with ID starting with 0dd4e2ef096bf3a63128e3c155bf65419860caebc8e81ef457184435ac81577f not found: ID does not exist" containerID="0dd4e2ef096bf3a63128e3c155bf65419860caebc8e81ef457184435ac81577f"
Aug 04 18:19:23.860033 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.859957    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd4e2ef096bf3a63128e3c155bf65419860caebc8e81ef457184435ac81577f"} err="failed to get container status \"0dd4e2ef096bf3a63128e3c155bf65419860caebc8e81ef457184435ac81577f\": rpc error: code = NotFound desc = could not find container \"0dd4e2ef096bf3a63128e3c155bf65419860caebc8e81ef457184435ac81577f\": container with ID starting with 0dd4e2ef096bf3a63128e3c155bf65419860caebc8e81ef457184435ac81577f not found: ID does not exist"
Aug 04 18:19:23.860033 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.859979    2566 scope.go:117] "RemoveContainer" containerID="17cc7a9a338d8c913349ccc41cce5e6c3876364068cc0cf23df963752f1e089a"
Aug 04 18:19:23.860466 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:19:23.860281    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17cc7a9a338d8c913349ccc41cce5e6c3876364068cc0cf23df963752f1e089a\": container with ID starting with 17cc7a9a338d8c913349ccc41cce5e6c3876364068cc0cf23df963752f1e089a not found: ID does not exist" containerID="17cc7a9a338d8c913349ccc41cce5e6c3876364068cc0cf23df963752f1e089a"
Aug 04 18:19:23.860567 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.860480    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17cc7a9a338d8c913349ccc41cce5e6c3876364068cc0cf23df963752f1e089a"} err="failed to get container status \"17cc7a9a338d8c913349ccc41cce5e6c3876364068cc0cf23df963752f1e089a\": rpc error: code = NotFound desc = could not find container \"17cc7a9a338d8c913349ccc41cce5e6c3876364068cc0cf23df963752f1e089a\": container with ID starting with 17cc7a9a338d8c913349ccc41cce5e6c3876364068cc0cf23df963752f1e089a not found: ID does not exist"
Aug 04 18:19:23.860567 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.860498    2566 scope.go:117] "RemoveContainer" containerID="5cba2d9e83bf779531c2e882376b789369397e891a0f9b323511d94d77638def"
Aug 04 18:19:23.860805 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:19:23.860783    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cba2d9e83bf779531c2e882376b789369397e891a0f9b323511d94d77638def\": container with ID starting with 5cba2d9e83bf779531c2e882376b789369397e891a0f9b323511d94d77638def not found: ID does not exist" containerID="5cba2d9e83bf779531c2e882376b789369397e891a0f9b323511d94d77638def"
Aug 04 18:19:23.860856 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.860811    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cba2d9e83bf779531c2e882376b789369397e891a0f9b323511d94d77638def"} err="failed to get container status \"5cba2d9e83bf779531c2e882376b789369397e891a0f9b323511d94d77638def\": rpc error: code = NotFound desc = could not find container \"5cba2d9e83bf779531c2e882376b789369397e891a0f9b323511d94d77638def\": container with ID starting with 5cba2d9e83bf779531c2e882376b789369397e891a0f9b323511d94d77638def not found: ID does not exist"
Aug 04 18:19:23.861245 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:23.861229    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-blrnm"]
Aug 04 18:19:24.013495 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:24.013463    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"]
Aug 04 18:19:24.014734 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:19:24.014705    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959496f4_2deb_495f_a8b3_0fe4aacc6634.slice/crio-579374bce86c59c45dc1f595535107f0f6d6ba9c4ae193037cafa5f8800f91f2 WatchSource:0}: Error finding container 579374bce86c59c45dc1f595535107f0f6d6ba9c4ae193037cafa5f8800f91f2: Status 404 returned error can't find the container with id 579374bce86c59c45dc1f595535107f0f6d6ba9c4ae193037cafa5f8800f91f2
Aug 04 18:19:24.832544 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:24.832505    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" event={"ID":"959496f4-2deb-495f-a8b3-0fe4aacc6634","Type":"ContainerStarted","Data":"f4d016e6d8c8e86589291bdd44174fbfaaf4219ecc420cb99ac1e5f62f1cd7af"}
Aug 04 18:19:24.833023 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:24.832567    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" event={"ID":"959496f4-2deb-495f-a8b3-0fe4aacc6634","Type":"ContainerStarted","Data":"579374bce86c59c45dc1f595535107f0f6d6ba9c4ae193037cafa5f8800f91f2"}
Aug 04 18:19:25.194006 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:25.193928    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2dfb04f-e78b-439d-8db7-6d9daf9e00de" path="/var/lib/kubelet/pods/f2dfb04f-e78b-439d-8db7-6d9daf9e00de/volumes"
Aug 04 18:19:25.810004 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:25.809977    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:19:25.810612 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:25.810578    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k" podUID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Aug 04 18:19:28.854864 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:28.854825    2566 generic.go:356] "Generic (PLEG): container finished" podID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerID="f4d016e6d8c8e86589291bdd44174fbfaaf4219ecc420cb99ac1e5f62f1cd7af" exitCode=0
Aug 04 18:19:28.855345 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:28.854902    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" event={"ID":"959496f4-2deb-495f-a8b3-0fe4aacc6634","Type":"ContainerDied","Data":"f4d016e6d8c8e86589291bdd44174fbfaaf4219ecc420cb99ac1e5f62f1cd7af"}
Aug 04 18:19:29.861500 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:29.861467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" event={"ID":"959496f4-2deb-495f-a8b3-0fe4aacc6634","Type":"ContainerStarted","Data":"1bc838b68a27fd96f22b2c286d16c8dbedb0788439bb2b841df91754ac2acec1"}
Aug 04 18:19:29.861905 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:29.861529    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" event={"ID":"959496f4-2deb-495f-a8b3-0fe4aacc6634","Type":"ContainerStarted","Data":"04bf92d4627c05a3562ad82c93f3253d7c9f5e6b8058d959ace5ebc2af2b87e3"}
Aug 04 18:19:29.861905 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:29.861808    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:29.861986 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:29.861964    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:29.863176 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:29.863149    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 04 18:19:29.883779 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:29.883735    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" podStartSLOduration=3.153836644 podStartE2EDuration="6.883720261s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.729883617s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:19:29.883514462 +0000 UTC m=+1169.174128668" watchObservedRunningTime="2026-08-04 18:19:29.883720261 +0000 UTC m=+1169.174334431"
Aug 04 18:19:30.866660 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:30.866624    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 04 18:19:35.811479 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:35.811454    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:19:35.871156 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:35.871127    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:19:35.871560 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:35.871532    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 04 18:19:45.872445 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:45.872403    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 04 18:19:51.270667 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:51.270636    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"]
Aug 04 18:19:51.271173 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:51.270858    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k" podUID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" containerName="kserve-container" containerID="cri-o://242ed36158f5202ac70c7ed199d0af6b49fce98501e3e4151d8f56141d760ce2" gracePeriod=30
Aug 04 18:19:51.271173 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:51.270912    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k" podUID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" containerName="kube-rbac-proxy" containerID="cri-o://f912b0e66a4b1547b55e587f5c752fdb7af21d7491715fe093f20b0dce95a978" gracePeriod=30
Aug 04 18:19:51.978204 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:51.978172    2566 generic.go:356] "Generic (PLEG): container finished" podID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" containerID="f912b0e66a4b1547b55e587f5c752fdb7af21d7491715fe093f20b0dce95a978" exitCode=2
Aug 04 18:19:51.978397 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:51.978254    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k" event={"ID":"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e","Type":"ContainerDied","Data":"f912b0e66a4b1547b55e587f5c752fdb7af21d7491715fe093f20b0dce95a978"}
Aug 04 18:19:52.750578 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.750551    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:19:52.904458 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.904422    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-cabundle-cert\") pod \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") "
Aug 04 18:19:52.904624 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.904516    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kserve-provision-location\") pod \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") "
Aug 04 18:19:52.904624 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.904552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzdsr\" (UniqueName: \"kubernetes.io/projected/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kube-api-access-qzdsr\") pod \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") "
Aug 04 18:19:52.904624 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.904604    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-proxy-tls\") pod \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") "
Aug 04 18:19:52.904752 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.904642    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kube-rbac-proxy-sar-config\") pod \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\" (UID: \"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e\") "
Aug 04 18:19:52.904819 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.904792    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" (UID: "a8c7b0ea-6691-4b63-8e35-13a7aa607f6e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:19:52.904945 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.904927    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:19:52.905101 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.905078    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" (UID: "a8c7b0ea-6691-4b63-8e35-13a7aa607f6e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:19:52.906630 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.906604    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kube-api-access-qzdsr" (OuterVolumeSpecName: "kube-api-access-qzdsr") pod "a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" (UID: "a8c7b0ea-6691-4b63-8e35-13a7aa607f6e"). InnerVolumeSpecName "kube-api-access-qzdsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:19:52.906720 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.906700    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" (UID: "a8c7b0ea-6691-4b63-8e35-13a7aa607f6e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:19:52.908378 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.908340    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" (UID: "a8c7b0ea-6691-4b63-8e35-13a7aa607f6e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:19:52.984503 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.984466    2566 generic.go:356] "Generic (PLEG): container finished" podID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" containerID="242ed36158f5202ac70c7ed199d0af6b49fce98501e3e4151d8f56141d760ce2" exitCode=0
Aug 04 18:19:52.984665 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.984530    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k" event={"ID":"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e","Type":"ContainerDied","Data":"242ed36158f5202ac70c7ed199d0af6b49fce98501e3e4151d8f56141d760ce2"}
Aug 04 18:19:52.984665 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.984544    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"
Aug 04 18:19:52.984665 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.984584    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k" event={"ID":"a8c7b0ea-6691-4b63-8e35-13a7aa607f6e","Type":"ContainerDied","Data":"531cb5c5ecacf75062e9a1c03f7c8e28883e713344334e009af84f5d4f614cd4"}
Aug 04 18:19:52.984665 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.984611    2566 scope.go:117] "RemoveContainer" containerID="f912b0e66a4b1547b55e587f5c752fdb7af21d7491715fe093f20b0dce95a978"
Aug 04 18:19:52.993941 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:52.993920    2566 scope.go:117] "RemoveContainer" containerID="242ed36158f5202ac70c7ed199d0af6b49fce98501e3e4151d8f56141d760ce2"
Aug 04 18:19:53.001616 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:53.001593    2566 scope.go:117] "RemoveContainer" containerID="962332098c3af1a912914981ae484e4fb22dd25e37af4ac334c0e59ec738dc2f"
Aug 04 18:19:53.005823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:53.005804    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:19:53.005904 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:53.005825    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzdsr\" (UniqueName: \"kubernetes.io/projected/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kube-api-access-qzdsr\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:19:53.005904 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:53.005835    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:19:53.005904 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:53.005844    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:19:53.008817 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:53.008791    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"]
Aug 04 18:19:53.009226 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:53.009207    2566 scope.go:117] "RemoveContainer" containerID="f912b0e66a4b1547b55e587f5c752fdb7af21d7491715fe093f20b0dce95a978"
Aug 04 18:19:53.009518 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:19:53.009501    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f912b0e66a4b1547b55e587f5c752fdb7af21d7491715fe093f20b0dce95a978\": container with ID starting with f912b0e66a4b1547b55e587f5c752fdb7af21d7491715fe093f20b0dce95a978 not found: ID does not exist" containerID="f912b0e66a4b1547b55e587f5c752fdb7af21d7491715fe093f20b0dce95a978"
Aug 04 18:19:53.009586 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:53.009531    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f912b0e66a4b1547b55e587f5c752fdb7af21d7491715fe093f20b0dce95a978"} err="failed to get container status \"f912b0e66a4b1547b55e587f5c752fdb7af21d7491715fe093f20b0dce95a978\": rpc error: code = NotFound desc = could not find container \"f912b0e66a4b1547b55e587f5c752fdb7af21d7491715fe093f20b0dce95a978\": container with ID starting with f912b0e66a4b1547b55e587f5c752fdb7af21d7491715fe093f20b0dce95a978 not found: ID does not exist"
Aug 04 18:19:53.009586 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:53.009555    2566 scope.go:117] "RemoveContainer" containerID="242ed36158f5202ac70c7ed199d0af6b49fce98501e3e4151d8f56141d760ce2"
Aug 04 18:19:53.009807 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:19:53.009790    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242ed36158f5202ac70c7ed199d0af6b49fce98501e3e4151d8f56141d760ce2\": container with ID starting with 242ed36158f5202ac70c7ed199d0af6b49fce98501e3e4151d8f56141d760ce2 not found: ID does not exist" containerID="242ed36158f5202ac70c7ed199d0af6b49fce98501e3e4151d8f56141d760ce2"
Aug 04 18:19:53.009850 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:53.009814    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242ed36158f5202ac70c7ed199d0af6b49fce98501e3e4151d8f56141d760ce2"} err="failed to get container status \"242ed36158f5202ac70c7ed199d0af6b49fce98501e3e4151d8f56141d760ce2\": rpc error: code = NotFound desc = could not find container \"242ed36158f5202ac70c7ed199d0af6b49fce98501e3e4151d8f56141d760ce2\": container with ID starting with 242ed36158f5202ac70c7ed199d0af6b49fce98501e3e4151d8f56141d760ce2 not found: ID does not exist"
Aug 04 18:19:53.009850 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:53.009833    2566 scope.go:117] "RemoveContainer" containerID="962332098c3af1a912914981ae484e4fb22dd25e37af4ac334c0e59ec738dc2f"
Aug 04 18:19:53.010042 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:19:53.010025    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962332098c3af1a912914981ae484e4fb22dd25e37af4ac334c0e59ec738dc2f\": container with ID starting with 962332098c3af1a912914981ae484e4fb22dd25e37af4ac334c0e59ec738dc2f not found: ID does not exist" containerID="962332098c3af1a912914981ae484e4fb22dd25e37af4ac334c0e59ec738dc2f"
Aug 04 18:19:53.010103 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:53.010050    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962332098c3af1a912914981ae484e4fb22dd25e37af4ac334c0e59ec738dc2f"} err="failed to get container status \"962332098c3af1a912914981ae484e4fb22dd25e37af4ac334c0e59ec738dc2f\": rpc error: code = NotFound desc = could not find container \"962332098c3af1a912914981ae484e4fb22dd25e37af4ac334c0e59ec738dc2f\": container with ID starting with 962332098c3af1a912914981ae484e4fb22dd25e37af4ac334c0e59ec738dc2f not found: ID does not exist"
Aug 04 18:19:53.014195 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:53.014173    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-8h78k"]
Aug 04 18:19:53.194631 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:53.194555    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" path="/var/lib/kubelet/pods/a8c7b0ea-6691-4b63-8e35-13a7aa607f6e/volumes"
Aug 04 18:19:55.871848 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:55.871808    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 04 18:19:56.459178 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.459148    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"]
Aug 04 18:19:56.459545 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.459531    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" containerName="kserve-container"
Aug 04 18:19:56.459601 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.459546    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" containerName="kserve-container"
Aug 04 18:19:56.459601 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.459563    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" containerName="storage-initializer"
Aug 04 18:19:56.459601 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.459569    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" containerName="storage-initializer"
Aug 04 18:19:56.459601 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.459579    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" containerName="kube-rbac-proxy"
Aug 04 18:19:56.459601 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.459584    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" containerName="kube-rbac-proxy"
Aug 04 18:19:56.459789 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.459632    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" containerName="kserve-container"
Aug 04 18:19:56.459789 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.459641    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c7b0ea-6691-4b63-8e35-13a7aa607f6e" containerName="kube-rbac-proxy"
Aug 04 18:19:56.462750 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.462731    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.465468 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.465448    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Aug 04 18:19:56.465468 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.465461    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Aug 04 18:19:56.475264 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.475245    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"]
Aug 04 18:19:56.640456 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.640425    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq84l\" (UniqueName: \"kubernetes.io/projected/9d21889a-cb50-48da-8257-b635d65b779b-kube-api-access-jq84l\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-jj7hc\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.640635 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.640473    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d21889a-cb50-48da-8257-b635d65b779b-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-jj7hc\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.640635 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.640501    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d21889a-cb50-48da-8257-b635d65b779b-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-jj7hc\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.640635 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.640572    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d21889a-cb50-48da-8257-b635d65b779b-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-jj7hc\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.640758 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.640635    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d21889a-cb50-48da-8257-b635d65b779b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-jj7hc\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.741558 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.741471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq84l\" (UniqueName: \"kubernetes.io/projected/9d21889a-cb50-48da-8257-b635d65b779b-kube-api-access-jq84l\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-jj7hc\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.741558 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.741520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d21889a-cb50-48da-8257-b635d65b779b-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-jj7hc\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.741558 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.741549    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d21889a-cb50-48da-8257-b635d65b779b-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-jj7hc\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.741852 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.741573    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d21889a-cb50-48da-8257-b635d65b779b-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-jj7hc\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.741852 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.741610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d21889a-cb50-48da-8257-b635d65b779b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-jj7hc\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.741979 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.741957    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d21889a-cb50-48da-8257-b635d65b779b-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-jj7hc\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.742224 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.742207    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d21889a-cb50-48da-8257-b635d65b779b-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-jj7hc\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.742283 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.742270    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d21889a-cb50-48da-8257-b635d65b779b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-jj7hc\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.744215 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.744188    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d21889a-cb50-48da-8257-b635d65b779b-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-jj7hc\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.752137 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.752109    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq84l\" (UniqueName: \"kubernetes.io/projected/9d21889a-cb50-48da-8257-b635d65b779b-kube-api-access-jq84l\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-jj7hc\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.773181 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.773152    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:19:56.911410 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:56.911383    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"]
Aug 04 18:19:56.913269 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:19:56.913239    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d21889a_cb50_48da_8257_b635d65b779b.slice/crio-75fac7b7aac0894fe4a3f8e96fcd82d71d615838daec5f265e699b5294639548 WatchSource:0}: Error finding container 75fac7b7aac0894fe4a3f8e96fcd82d71d615838daec5f265e699b5294639548: Status 404 returned error can't find the container with id 75fac7b7aac0894fe4a3f8e96fcd82d71d615838daec5f265e699b5294639548
Aug 04 18:19:57.005944 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:57.005848    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc" event={"ID":"9d21889a-cb50-48da-8257-b635d65b779b","Type":"ContainerStarted","Data":"462827945e761b768fafb931925dee6208192948d4b4879850715a38ec1ca110"}
Aug 04 18:19:57.005944 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:19:57.005923    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc" event={"ID":"9d21889a-cb50-48da-8257-b635d65b779b","Type":"ContainerStarted","Data":"75fac7b7aac0894fe4a3f8e96fcd82d71d615838daec5f265e699b5294639548"}
Aug 04 18:20:01.255264 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:01.255235    2566 scope.go:117] "RemoveContainer" containerID="7649a2364a43cda604be4d413e999fdeb5574cb5906d3ca7be12fc4b667d4cc1"
Aug 04 18:20:01.263369 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:01.263331    2566 scope.go:117] "RemoveContainer" containerID="bad314ea4729d775977633086926d41c3bbccb8778e6c7a2ab8715a7d183e00d"
Aug 04 18:20:01.270735 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:01.270715    2566 scope.go:117] "RemoveContainer" containerID="2219777eb7f93da198db09aac3f57e8c81174f3dc10ccc748db9af75f753b3b9"
Aug 04 18:20:01.277826 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:01.277809    2566 scope.go:117] "RemoveContainer" containerID="10a69171e313da94bc522cee19e008688ce9e49b2e7e4494678ec6d7c1b23f98"
Aug 04 18:20:05.872621 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:05.872582    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 04 18:20:15.872304 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:15.872268    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 04 18:20:25.872639 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:25.872589    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 04 18:20:27.154403 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:27.154370    2566 generic.go:356] "Generic (PLEG): container finished" podID="9d21889a-cb50-48da-8257-b635d65b779b" containerID="462827945e761b768fafb931925dee6208192948d4b4879850715a38ec1ca110" exitCode=0
Aug 04 18:20:27.154403 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:27.154389    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc" event={"ID":"9d21889a-cb50-48da-8257-b635d65b779b","Type":"ContainerDied","Data":"462827945e761b768fafb931925dee6208192948d4b4879850715a38ec1ca110"}
Aug 04 18:20:28.161745 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:28.161712    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc" event={"ID":"9d21889a-cb50-48da-8257-b635d65b779b","Type":"ContainerStarted","Data":"79d9514c8eacf0675cd3426553cfd8d0eadb86d80741aec0b78ef5f3d082d27e"}
Aug 04 18:20:28.162147 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:28.161768    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc" event={"ID":"9d21889a-cb50-48da-8257-b635d65b779b","Type":"ContainerStarted","Data":"6b01c96934ce40b3153b6d5ef8fe5872891b76f328db1bc5fde1b1c6bb62a6c2"}
Aug 04 18:20:28.162147 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:28.161957    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:20:29.167103 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:29.167060    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:20:35.178101 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:35.178073    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:20:35.205301 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:35.205239    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc" podStartSLOduration=9.646014231 podStartE2EDuration="39.205222593s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.559208362s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:20:28.186845182 +0000 UTC m=+1227.477459376" watchObservedRunningTime="2026-08-04 18:20:35.205222593 +0000 UTC m=+1234.495836764"
Aug 04 18:20:35.872011 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:35.871971    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 04 18:20:45.872482 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:45.872450    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:20:53.605124 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:53.605048    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"]
Aug 04 18:20:53.605564 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:53.605277    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kserve-container" containerID="cri-o://04bf92d4627c05a3562ad82c93f3253d7c9f5e6b8058d959ace5ebc2af2b87e3" gracePeriod=30
Aug 04 18:20:53.605564 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:53.605328    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kube-rbac-proxy" containerID="cri-o://1bc838b68a27fd96f22b2c286d16c8dbedb0788439bb2b841df91754ac2acec1" gracePeriod=30
Aug 04 18:20:54.290058 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:54.290026    2566 generic.go:356] "Generic (PLEG): container finished" podID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerID="1bc838b68a27fd96f22b2c286d16c8dbedb0788439bb2b841df91754ac2acec1" exitCode=2
Aug 04 18:20:54.290238 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:54.290102    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" event={"ID":"959496f4-2deb-495f-a8b3-0fe4aacc6634","Type":"ContainerDied","Data":"1bc838b68a27fd96f22b2c286d16c8dbedb0788439bb2b841df91754ac2acec1"}
Aug 04 18:20:55.867641 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:55.867600    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Aug 04 18:20:55.871961 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:55.871937    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Aug 04 18:20:58.844642 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:58.844617    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:20:58.911171 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:58.911135    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmb4n\" (UniqueName: \"kubernetes.io/projected/959496f4-2deb-495f-a8b3-0fe4aacc6634-kube-api-access-rmb4n\") pod \"959496f4-2deb-495f-a8b3-0fe4aacc6634\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") "
Aug 04 18:20:58.911171 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:58.911174    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/959496f4-2deb-495f-a8b3-0fe4aacc6634-kube-rbac-proxy-sar-config\") pod \"959496f4-2deb-495f-a8b3-0fe4aacc6634\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") "
Aug 04 18:20:58.911475 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:58.911222    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/959496f4-2deb-495f-a8b3-0fe4aacc6634-proxy-tls\") pod \"959496f4-2deb-495f-a8b3-0fe4aacc6634\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") "
Aug 04 18:20:58.911475 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:58.911297    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/959496f4-2deb-495f-a8b3-0fe4aacc6634-kserve-provision-location\") pod \"959496f4-2deb-495f-a8b3-0fe4aacc6634\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") "
Aug 04 18:20:58.911475 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:58.911328    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/959496f4-2deb-495f-a8b3-0fe4aacc6634-cabundle-cert\") pod \"959496f4-2deb-495f-a8b3-0fe4aacc6634\" (UID: \"959496f4-2deb-495f-a8b3-0fe4aacc6634\") "
Aug 04 18:20:58.911689 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:58.911658    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959496f4-2deb-495f-a8b3-0fe4aacc6634-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "959496f4-2deb-495f-a8b3-0fe4aacc6634" (UID: "959496f4-2deb-495f-a8b3-0fe4aacc6634"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:20:58.911824 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:58.911608    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959496f4-2deb-495f-a8b3-0fe4aacc6634-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "959496f4-2deb-495f-a8b3-0fe4aacc6634" (UID: "959496f4-2deb-495f-a8b3-0fe4aacc6634"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:20:58.911824 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:58.911778    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959496f4-2deb-495f-a8b3-0fe4aacc6634-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "959496f4-2deb-495f-a8b3-0fe4aacc6634" (UID: "959496f4-2deb-495f-a8b3-0fe4aacc6634"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:20:58.913322 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:58.913300    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/959496f4-2deb-495f-a8b3-0fe4aacc6634-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "959496f4-2deb-495f-a8b3-0fe4aacc6634" (UID: "959496f4-2deb-495f-a8b3-0fe4aacc6634"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:20:58.913436 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:58.913345    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959496f4-2deb-495f-a8b3-0fe4aacc6634-kube-api-access-rmb4n" (OuterVolumeSpecName: "kube-api-access-rmb4n") pod "959496f4-2deb-495f-a8b3-0fe4aacc6634" (UID: "959496f4-2deb-495f-a8b3-0fe4aacc6634"). InnerVolumeSpecName "kube-api-access-rmb4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:20:59.012071 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.011986    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/959496f4-2deb-495f-a8b3-0fe4aacc6634-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:20:59.012071 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.012017    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/959496f4-2deb-495f-a8b3-0fe4aacc6634-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:20:59.012071 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.012027    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmb4n\" (UniqueName: \"kubernetes.io/projected/959496f4-2deb-495f-a8b3-0fe4aacc6634-kube-api-access-rmb4n\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:20:59.012071 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.012036    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/959496f4-2deb-495f-a8b3-0fe4aacc6634-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:20:59.012071 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.012048    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/959496f4-2deb-495f-a8b3-0fe4aacc6634-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:20:59.314618 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.314587    2566 generic.go:356] "Generic (PLEG): container finished" podID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerID="04bf92d4627c05a3562ad82c93f3253d7c9f5e6b8058d959ace5ebc2af2b87e3" exitCode=0
Aug 04 18:20:59.314798 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.314645    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" event={"ID":"959496f4-2deb-495f-a8b3-0fe4aacc6634","Type":"ContainerDied","Data":"04bf92d4627c05a3562ad82c93f3253d7c9f5e6b8058d959ace5ebc2af2b87e3"}
Aug 04 18:20:59.314798 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.314659    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"
Aug 04 18:20:59.314798 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.314693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm" event={"ID":"959496f4-2deb-495f-a8b3-0fe4aacc6634","Type":"ContainerDied","Data":"579374bce86c59c45dc1f595535107f0f6d6ba9c4ae193037cafa5f8800f91f2"}
Aug 04 18:20:59.314798 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.314722    2566 scope.go:117] "RemoveContainer" containerID="1bc838b68a27fd96f22b2c286d16c8dbedb0788439bb2b841df91754ac2acec1"
Aug 04 18:20:59.325407 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.325385    2566 scope.go:117] "RemoveContainer" containerID="04bf92d4627c05a3562ad82c93f3253d7c9f5e6b8058d959ace5ebc2af2b87e3"
Aug 04 18:20:59.333152 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.333135    2566 scope.go:117] "RemoveContainer" containerID="f4d016e6d8c8e86589291bdd44174fbfaaf4219ecc420cb99ac1e5f62f1cd7af"
Aug 04 18:20:59.338743 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.338715    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"]
Aug 04 18:20:59.340972 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.340956    2566 scope.go:117] "RemoveContainer" containerID="1bc838b68a27fd96f22b2c286d16c8dbedb0788439bb2b841df91754ac2acec1"
Aug 04 18:20:59.341281 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:20:59.341258    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc838b68a27fd96f22b2c286d16c8dbedb0788439bb2b841df91754ac2acec1\": container with ID starting with 1bc838b68a27fd96f22b2c286d16c8dbedb0788439bb2b841df91754ac2acec1 not found: ID does not exist" containerID="1bc838b68a27fd96f22b2c286d16c8dbedb0788439bb2b841df91754ac2acec1"
Aug 04 18:20:59.342065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.341292    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc838b68a27fd96f22b2c286d16c8dbedb0788439bb2b841df91754ac2acec1"} err="failed to get container status \"1bc838b68a27fd96f22b2c286d16c8dbedb0788439bb2b841df91754ac2acec1\": rpc error: code = NotFound desc = could not find container \"1bc838b68a27fd96f22b2c286d16c8dbedb0788439bb2b841df91754ac2acec1\": container with ID starting with 1bc838b68a27fd96f22b2c286d16c8dbedb0788439bb2b841df91754ac2acec1 not found: ID does not exist"
Aug 04 18:20:59.342065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.341310    2566 scope.go:117] "RemoveContainer" containerID="04bf92d4627c05a3562ad82c93f3253d7c9f5e6b8058d959ace5ebc2af2b87e3"
Aug 04 18:20:59.342065 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:20:59.341796    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04bf92d4627c05a3562ad82c93f3253d7c9f5e6b8058d959ace5ebc2af2b87e3\": container with ID starting with 04bf92d4627c05a3562ad82c93f3253d7c9f5e6b8058d959ace5ebc2af2b87e3 not found: ID does not exist" containerID="04bf92d4627c05a3562ad82c93f3253d7c9f5e6b8058d959ace5ebc2af2b87e3"
Aug 04 18:20:59.342065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.341828    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04bf92d4627c05a3562ad82c93f3253d7c9f5e6b8058d959ace5ebc2af2b87e3"} err="failed to get container status \"04bf92d4627c05a3562ad82c93f3253d7c9f5e6b8058d959ace5ebc2af2b87e3\": rpc error: code = NotFound desc = could not find container \"04bf92d4627c05a3562ad82c93f3253d7c9f5e6b8058d959ace5ebc2af2b87e3\": container with ID starting with 04bf92d4627c05a3562ad82c93f3253d7c9f5e6b8058d959ace5ebc2af2b87e3 not found: ID does not exist"
Aug 04 18:20:59.342065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.341848    2566 scope.go:117] "RemoveContainer" containerID="f4d016e6d8c8e86589291bdd44174fbfaaf4219ecc420cb99ac1e5f62f1cd7af"
Aug 04 18:20:59.342369 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:20:59.342168    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d016e6d8c8e86589291bdd44174fbfaaf4219ecc420cb99ac1e5f62f1cd7af\": container with ID starting with f4d016e6d8c8e86589291bdd44174fbfaaf4219ecc420cb99ac1e5f62f1cd7af not found: ID does not exist" containerID="f4d016e6d8c8e86589291bdd44174fbfaaf4219ecc420cb99ac1e5f62f1cd7af"
Aug 04 18:20:59.342369 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.342197    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d016e6d8c8e86589291bdd44174fbfaaf4219ecc420cb99ac1e5f62f1cd7af"} err="failed to get container status \"f4d016e6d8c8e86589291bdd44174fbfaaf4219ecc420cb99ac1e5f62f1cd7af\": rpc error: code = NotFound desc = could not find container \"f4d016e6d8c8e86589291bdd44174fbfaaf4219ecc420cb99ac1e5f62f1cd7af\": container with ID starting with f4d016e6d8c8e86589291bdd44174fbfaaf4219ecc420cb99ac1e5f62f1cd7af not found: ID does not exist"
Aug 04 18:20:59.343933 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:20:59.343917    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fdxbm"]
Aug 04 18:21:01.194445 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:21:01.194416    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" path="/var/lib/kubelet/pods/959496f4-2deb-495f-a8b3-0fe4aacc6634/volumes"
Aug 04 18:21:58.192870 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:21:58.192842    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:22:09.229638 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:09.229578    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc" podUID="9d21889a-cb50-48da-8257-b635d65b779b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.42:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Aug 04 18:22:12.856655 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:12.856609    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"]
Aug 04 18:22:12.857124 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:12.856945    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc" podUID="9d21889a-cb50-48da-8257-b635d65b779b" containerName="kube-rbac-proxy" containerID="cri-o://79d9514c8eacf0675cd3426553cfd8d0eadb86d80741aec0b78ef5f3d082d27e" gracePeriod=30
Aug 04 18:22:12.857124 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:12.856945    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc" podUID="9d21889a-cb50-48da-8257-b635d65b779b" containerName="kserve-container" containerID="cri-o://6b01c96934ce40b3153b6d5ef8fe5872891b76f328db1bc5fde1b1c6bb62a6c2" gracePeriod=30
Aug 04 18:22:13.648757 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:13.648722    2566 generic.go:356] "Generic (PLEG): container finished" podID="9d21889a-cb50-48da-8257-b635d65b779b" containerID="79d9514c8eacf0675cd3426553cfd8d0eadb86d80741aec0b78ef5f3d082d27e" exitCode=2
Aug 04 18:22:13.648935 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:13.648802    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc" event={"ID":"9d21889a-cb50-48da-8257-b635d65b779b","Type":"ContainerDied","Data":"79d9514c8eacf0675cd3426553cfd8d0eadb86d80741aec0b78ef5f3d082d27e"}
Aug 04 18:22:14.308140 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.308107    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:22:14.409975 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.409886    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq84l\" (UniqueName: \"kubernetes.io/projected/9d21889a-cb50-48da-8257-b635d65b779b-kube-api-access-jq84l\") pod \"9d21889a-cb50-48da-8257-b635d65b779b\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") "
Aug 04 18:22:14.409975 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.409929    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d21889a-cb50-48da-8257-b635d65b779b-proxy-tls\") pod \"9d21889a-cb50-48da-8257-b635d65b779b\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") "
Aug 04 18:22:14.409975 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.409947    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d21889a-cb50-48da-8257-b635d65b779b-kube-rbac-proxy-sar-config\") pod \"9d21889a-cb50-48da-8257-b635d65b779b\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") "
Aug 04 18:22:14.409975 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.409968    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d21889a-cb50-48da-8257-b635d65b779b-kserve-provision-location\") pod \"9d21889a-cb50-48da-8257-b635d65b779b\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") "
Aug 04 18:22:14.410337 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.410043    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d21889a-cb50-48da-8257-b635d65b779b-cabundle-cert\") pod \"9d21889a-cb50-48da-8257-b635d65b779b\" (UID: \"9d21889a-cb50-48da-8257-b635d65b779b\") "
Aug 04 18:22:14.410452 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.410423    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d21889a-cb50-48da-8257-b635d65b779b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9d21889a-cb50-48da-8257-b635d65b779b" (UID: "9d21889a-cb50-48da-8257-b635d65b779b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:22:14.410506 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.410429    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d21889a-cb50-48da-8257-b635d65b779b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9d21889a-cb50-48da-8257-b635d65b779b" (UID: "9d21889a-cb50-48da-8257-b635d65b779b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:22:14.412043 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.412023    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d21889a-cb50-48da-8257-b635d65b779b-kube-api-access-jq84l" (OuterVolumeSpecName: "kube-api-access-jq84l") pod "9d21889a-cb50-48da-8257-b635d65b779b" (UID: "9d21889a-cb50-48da-8257-b635d65b779b"). InnerVolumeSpecName "kube-api-access-jq84l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:22:14.412121 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.412078    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d21889a-cb50-48da-8257-b635d65b779b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d21889a-cb50-48da-8257-b635d65b779b" (UID: "9d21889a-cb50-48da-8257-b635d65b779b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:22:14.416700 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.416661    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d21889a-cb50-48da-8257-b635d65b779b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d21889a-cb50-48da-8257-b635d65b779b" (UID: "9d21889a-cb50-48da-8257-b635d65b779b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:22:14.511057 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.511023    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d21889a-cb50-48da-8257-b635d65b779b-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:22:14.511057 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.511051    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq84l\" (UniqueName: \"kubernetes.io/projected/9d21889a-cb50-48da-8257-b635d65b779b-kube-api-access-jq84l\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:22:14.511057 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.511064    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d21889a-cb50-48da-8257-b635d65b779b-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:22:14.511300 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.511073    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d21889a-cb50-48da-8257-b635d65b779b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:22:14.511300 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.511081    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d21889a-cb50-48da-8257-b635d65b779b-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:22:14.654947 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.654913    2566 generic.go:356] "Generic (PLEG): container finished" podID="9d21889a-cb50-48da-8257-b635d65b779b" containerID="6b01c96934ce40b3153b6d5ef8fe5872891b76f328db1bc5fde1b1c6bb62a6c2" exitCode=0
Aug 04 18:22:14.655115 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.654971    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"
Aug 04 18:22:14.655115 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.654997    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc" event={"ID":"9d21889a-cb50-48da-8257-b635d65b779b","Type":"ContainerDied","Data":"6b01c96934ce40b3153b6d5ef8fe5872891b76f328db1bc5fde1b1c6bb62a6c2"}
Aug 04 18:22:14.655115 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.655063    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc" event={"ID":"9d21889a-cb50-48da-8257-b635d65b779b","Type":"ContainerDied","Data":"75fac7b7aac0894fe4a3f8e96fcd82d71d615838daec5f265e699b5294639548"}
Aug 04 18:22:14.655115 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.655090    2566 scope.go:117] "RemoveContainer" containerID="79d9514c8eacf0675cd3426553cfd8d0eadb86d80741aec0b78ef5f3d082d27e"
Aug 04 18:22:14.664137 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.664120    2566 scope.go:117] "RemoveContainer" containerID="6b01c96934ce40b3153b6d5ef8fe5872891b76f328db1bc5fde1b1c6bb62a6c2"
Aug 04 18:22:14.671580 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.671563    2566 scope.go:117] "RemoveContainer" containerID="462827945e761b768fafb931925dee6208192948d4b4879850715a38ec1ca110"
Aug 04 18:22:14.678941 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.678924    2566 scope.go:117] "RemoveContainer" containerID="79d9514c8eacf0675cd3426553cfd8d0eadb86d80741aec0b78ef5f3d082d27e"
Aug 04 18:22:14.679206 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:22:14.679171    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d9514c8eacf0675cd3426553cfd8d0eadb86d80741aec0b78ef5f3d082d27e\": container with ID starting with 79d9514c8eacf0675cd3426553cfd8d0eadb86d80741aec0b78ef5f3d082d27e not found: ID does not exist" containerID="79d9514c8eacf0675cd3426553cfd8d0eadb86d80741aec0b78ef5f3d082d27e"
Aug 04 18:22:14.679299 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.679207    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d9514c8eacf0675cd3426553cfd8d0eadb86d80741aec0b78ef5f3d082d27e"} err="failed to get container status \"79d9514c8eacf0675cd3426553cfd8d0eadb86d80741aec0b78ef5f3d082d27e\": rpc error: code = NotFound desc = could not find container \"79d9514c8eacf0675cd3426553cfd8d0eadb86d80741aec0b78ef5f3d082d27e\": container with ID starting with 79d9514c8eacf0675cd3426553cfd8d0eadb86d80741aec0b78ef5f3d082d27e not found: ID does not exist"
Aug 04 18:22:14.679299 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.679226    2566 scope.go:117] "RemoveContainer" containerID="6b01c96934ce40b3153b6d5ef8fe5872891b76f328db1bc5fde1b1c6bb62a6c2"
Aug 04 18:22:14.679434 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:22:14.679424    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b01c96934ce40b3153b6d5ef8fe5872891b76f328db1bc5fde1b1c6bb62a6c2\": container with ID starting with 6b01c96934ce40b3153b6d5ef8fe5872891b76f328db1bc5fde1b1c6bb62a6c2 not found: ID does not exist" containerID="6b01c96934ce40b3153b6d5ef8fe5872891b76f328db1bc5fde1b1c6bb62a6c2"
Aug 04 18:22:14.679475 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.679442    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b01c96934ce40b3153b6d5ef8fe5872891b76f328db1bc5fde1b1c6bb62a6c2"} err="failed to get container status \"6b01c96934ce40b3153b6d5ef8fe5872891b76f328db1bc5fde1b1c6bb62a6c2\": rpc error: code = NotFound desc = could not find container \"6b01c96934ce40b3153b6d5ef8fe5872891b76f328db1bc5fde1b1c6bb62a6c2\": container with ID starting with 6b01c96934ce40b3153b6d5ef8fe5872891b76f328db1bc5fde1b1c6bb62a6c2 not found: ID does not exist"
Aug 04 18:22:14.679475 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.679455    2566 scope.go:117] "RemoveContainer" containerID="462827945e761b768fafb931925dee6208192948d4b4879850715a38ec1ca110"
Aug 04 18:22:14.679704 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:22:14.679687    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462827945e761b768fafb931925dee6208192948d4b4879850715a38ec1ca110\": container with ID starting with 462827945e761b768fafb931925dee6208192948d4b4879850715a38ec1ca110 not found: ID does not exist" containerID="462827945e761b768fafb931925dee6208192948d4b4879850715a38ec1ca110"
Aug 04 18:22:14.679745 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.679706    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462827945e761b768fafb931925dee6208192948d4b4879850715a38ec1ca110"} err="failed to get container status \"462827945e761b768fafb931925dee6208192948d4b4879850715a38ec1ca110\": rpc error: code = NotFound desc = could not find container \"462827945e761b768fafb931925dee6208192948d4b4879850715a38ec1ca110\": container with ID starting with 462827945e761b768fafb931925dee6208192948d4b4879850715a38ec1ca110 not found: ID does not exist"
Aug 04 18:22:14.683698 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.683675    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"]
Aug 04 18:22:14.689819 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:14.689798    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-jj7hc"]
Aug 04 18:22:15.194163 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:15.194131    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d21889a-cb50-48da-8257-b635d65b779b" path="/var/lib/kubelet/pods/9d21889a-cb50-48da-8257-b635d65b779b/volumes"
Aug 04 18:22:18.100884 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.100850    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"]
Aug 04 18:22:18.101254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.101197    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kube-rbac-proxy"
Aug 04 18:22:18.101254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.101209    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kube-rbac-proxy"
Aug 04 18:22:18.101254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.101222    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="storage-initializer"
Aug 04 18:22:18.101254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.101228    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="storage-initializer"
Aug 04 18:22:18.101254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.101235    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kserve-container"
Aug 04 18:22:18.101254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.101240    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kserve-container"
Aug 04 18:22:18.101254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.101247    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d21889a-cb50-48da-8257-b635d65b779b" containerName="kserve-container"
Aug 04 18:22:18.101254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.101252    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d21889a-cb50-48da-8257-b635d65b779b" containerName="kserve-container"
Aug 04 18:22:18.101254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.101259    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d21889a-cb50-48da-8257-b635d65b779b" containerName="kube-rbac-proxy"
Aug 04 18:22:18.101557 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.101264    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d21889a-cb50-48da-8257-b635d65b779b" containerName="kube-rbac-proxy"
Aug 04 18:22:18.101557 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.101278    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d21889a-cb50-48da-8257-b635d65b779b" containerName="storage-initializer"
Aug 04 18:22:18.101557 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.101283    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d21889a-cb50-48da-8257-b635d65b779b" containerName="storage-initializer"
Aug 04 18:22:18.101557 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.101342    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kserve-container"
Aug 04 18:22:18.101557 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.101366    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="959496f4-2deb-495f-a8b3-0fe4aacc6634" containerName="kube-rbac-proxy"
Aug 04 18:22:18.101557 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.101377    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d21889a-cb50-48da-8257-b635d65b779b" containerName="kserve-container"
Aug 04 18:22:18.101557 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.101382    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d21889a-cb50-48da-8257-b635d65b779b" containerName="kube-rbac-proxy"
Aug 04 18:22:18.106454 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.106433    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.110427 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.110404    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:22:18.110699 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.110677    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:22:18.111464 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.111450    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:22:18.111564 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.111499    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Aug 04 18:22:18.111564 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.111559    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Aug 04 18:22:18.111666 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.111567    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:22:18.111666 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.111580    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:22:18.132888 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.132856    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"]
Aug 04 18:22:18.144056 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.144026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12c30a4-8d8a-41aa-868a-ea19822800fa-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.144240 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.144091    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12c30a4-8d8a-41aa-868a-ea19822800fa-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.144240 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.144139    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5khx\" (UniqueName: \"kubernetes.io/projected/d12c30a4-8d8a-41aa-868a-ea19822800fa-kube-api-access-r5khx\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.144240 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.144198    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12c30a4-8d8a-41aa-868a-ea19822800fa-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.144440 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.144301    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12c30a4-8d8a-41aa-868a-ea19822800fa-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.245404 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.245338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5khx\" (UniqueName: \"kubernetes.io/projected/d12c30a4-8d8a-41aa-868a-ea19822800fa-kube-api-access-r5khx\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.245575 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.245423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12c30a4-8d8a-41aa-868a-ea19822800fa-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.245575 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.245509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12c30a4-8d8a-41aa-868a-ea19822800fa-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.245575 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.245545    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12c30a4-8d8a-41aa-868a-ea19822800fa-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.245690 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.245602    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12c30a4-8d8a-41aa-868a-ea19822800fa-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.246016 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.245993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12c30a4-8d8a-41aa-868a-ea19822800fa-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.246258 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.246242    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12c30a4-8d8a-41aa-868a-ea19822800fa-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.246294 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.246241    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12c30a4-8d8a-41aa-868a-ea19822800fa-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.247984 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.247965    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12c30a4-8d8a-41aa-868a-ea19822800fa-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.256662 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.256633    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5khx\" (UniqueName: \"kubernetes.io/projected/d12c30a4-8d8a-41aa-868a-ea19822800fa-kube-api-access-r5khx\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.417092 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.417008    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:18.561180 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.561154    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"]
Aug 04 18:22:18.563520 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:22:18.563489    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12c30a4_8d8a_41aa_868a_ea19822800fa.slice/crio-899104a3c9054bbe0a6afa1fb8221a3667ac47ed477161d32d7901546791db96 WatchSource:0}: Error finding container 899104a3c9054bbe0a6afa1fb8221a3667ac47ed477161d32d7901546791db96: Status 404 returned error can't find the container with id 899104a3c9054bbe0a6afa1fb8221a3667ac47ed477161d32d7901546791db96
Aug 04 18:22:18.675934 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.675856    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r" event={"ID":"d12c30a4-8d8a-41aa-868a-ea19822800fa","Type":"ContainerStarted","Data":"36eb4c8c7e70e193ba6ca70d71b4847fdde4591b471e74dd4c639339e99e252e"}
Aug 04 18:22:18.675934 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:18.675918    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r" event={"ID":"d12c30a4-8d8a-41aa-868a-ea19822800fa","Type":"ContainerStarted","Data":"899104a3c9054bbe0a6afa1fb8221a3667ac47ed477161d32d7901546791db96"}
Aug 04 18:22:38.776928 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:38.776900    2566 generic.go:356] "Generic (PLEG): container finished" podID="d12c30a4-8d8a-41aa-868a-ea19822800fa" containerID="36eb4c8c7e70e193ba6ca70d71b4847fdde4591b471e74dd4c639339e99e252e" exitCode=0
Aug 04 18:22:38.777286 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:38.776957    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r" event={"ID":"d12c30a4-8d8a-41aa-868a-ea19822800fa","Type":"ContainerDied","Data":"36eb4c8c7e70e193ba6ca70d71b4847fdde4591b471e74dd4c639339e99e252e"}
Aug 04 18:22:39.783684 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:39.783651    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r" event={"ID":"d12c30a4-8d8a-41aa-868a-ea19822800fa","Type":"ContainerStarted","Data":"52a7296b4e83ff3dffda055923e3780566f35b2dd903ddf5ed1e1774e011cb03"}
Aug 04 18:22:39.784046 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:39.783710    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r" event={"ID":"d12c30a4-8d8a-41aa-868a-ea19822800fa","Type":"ContainerStarted","Data":"1b87bb44d77f265d7dc7fe7901517db7cfe56816cfcf3ca0632029a5b8eeea5f"}
Aug 04 18:22:39.784046 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:39.783897    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:39.809612 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:39.809563    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r" podStartSLOduration=1.666666092 podStartE2EDuration="21.809546624s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.142880532s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:22:39.808069308 +0000 UTC m=+1359.098683534" watchObservedRunningTime="2026-08-04 18:22:39.809546624 +0000 UTC m=+1359.100160795"
Aug 04 18:22:40.795458 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:40.795424    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:22:46.806234 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:22:46.806209    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:24:08.192553 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:08.192522    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:24:19.230599 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:19.230549    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r" podUID="d12c30a4-8d8a-41aa-868a-ea19822800fa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.43:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Aug 04 18:24:24.985433 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:24.985398    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"]
Aug 04 18:24:24.985825 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:24.985666    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r" podUID="d12c30a4-8d8a-41aa-868a-ea19822800fa" containerName="kserve-container" containerID="cri-o://1b87bb44d77f265d7dc7fe7901517db7cfe56816cfcf3ca0632029a5b8eeea5f" gracePeriod=30
Aug 04 18:24:24.985825 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:24.985679    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r" podUID="d12c30a4-8d8a-41aa-868a-ea19822800fa" containerName="kube-rbac-proxy" containerID="cri-o://52a7296b4e83ff3dffda055923e3780566f35b2dd903ddf5ed1e1774e011cb03" gracePeriod=30
Aug 04 18:24:25.273184 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:25.273089    2566 generic.go:356] "Generic (PLEG): container finished" podID="d12c30a4-8d8a-41aa-868a-ea19822800fa" containerID="52a7296b4e83ff3dffda055923e3780566f35b2dd903ddf5ed1e1774e011cb03" exitCode=2
Aug 04 18:24:25.273184 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:25.273135    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r" event={"ID":"d12c30a4-8d8a-41aa-868a-ea19822800fa","Type":"ContainerDied","Data":"52a7296b4e83ff3dffda055923e3780566f35b2dd903ddf5ed1e1774e011cb03"}
Aug 04 18:24:26.279668 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.279635    2566 generic.go:356] "Generic (PLEG): container finished" podID="d12c30a4-8d8a-41aa-868a-ea19822800fa" containerID="1b87bb44d77f265d7dc7fe7901517db7cfe56816cfcf3ca0632029a5b8eeea5f" exitCode=0
Aug 04 18:24:26.280001 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.279708    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r" event={"ID":"d12c30a4-8d8a-41aa-868a-ea19822800fa","Type":"ContainerDied","Data":"1b87bb44d77f265d7dc7fe7901517db7cfe56816cfcf3ca0632029a5b8eeea5f"}
Aug 04 18:24:26.825147 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.825125    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:24:26.895020 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.894991    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5khx\" (UniqueName: \"kubernetes.io/projected/d12c30a4-8d8a-41aa-868a-ea19822800fa-kube-api-access-r5khx\") pod \"d12c30a4-8d8a-41aa-868a-ea19822800fa\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") "
Aug 04 18:24:26.895206 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.895031    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12c30a4-8d8a-41aa-868a-ea19822800fa-kserve-provision-location\") pod \"d12c30a4-8d8a-41aa-868a-ea19822800fa\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") "
Aug 04 18:24:26.895206 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.895051    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12c30a4-8d8a-41aa-868a-ea19822800fa-cabundle-cert\") pod \"d12c30a4-8d8a-41aa-868a-ea19822800fa\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") "
Aug 04 18:24:26.895206 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.895090    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12c30a4-8d8a-41aa-868a-ea19822800fa-kube-rbac-proxy-sar-config\") pod \"d12c30a4-8d8a-41aa-868a-ea19822800fa\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") "
Aug 04 18:24:26.895206 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.895134    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12c30a4-8d8a-41aa-868a-ea19822800fa-proxy-tls\") pod \"d12c30a4-8d8a-41aa-868a-ea19822800fa\" (UID: \"d12c30a4-8d8a-41aa-868a-ea19822800fa\") "
Aug 04 18:24:26.895493 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.895471    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12c30a4-8d8a-41aa-868a-ea19822800fa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d12c30a4-8d8a-41aa-868a-ea19822800fa" (UID: "d12c30a4-8d8a-41aa-868a-ea19822800fa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:24:26.895561 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.895508    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12c30a4-8d8a-41aa-868a-ea19822800fa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d12c30a4-8d8a-41aa-868a-ea19822800fa" (UID: "d12c30a4-8d8a-41aa-868a-ea19822800fa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:24:26.897256 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.897228    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12c30a4-8d8a-41aa-868a-ea19822800fa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d12c30a4-8d8a-41aa-868a-ea19822800fa" (UID: "d12c30a4-8d8a-41aa-868a-ea19822800fa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:24:26.897256 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.897236    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12c30a4-8d8a-41aa-868a-ea19822800fa-kube-api-access-r5khx" (OuterVolumeSpecName: "kube-api-access-r5khx") pod "d12c30a4-8d8a-41aa-868a-ea19822800fa" (UID: "d12c30a4-8d8a-41aa-868a-ea19822800fa"). InnerVolumeSpecName "kube-api-access-r5khx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:24:26.899919 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.899893    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12c30a4-8d8a-41aa-868a-ea19822800fa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d12c30a4-8d8a-41aa-868a-ea19822800fa" (UID: "d12c30a4-8d8a-41aa-868a-ea19822800fa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:24:26.996191 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.996092    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5khx\" (UniqueName: \"kubernetes.io/projected/d12c30a4-8d8a-41aa-868a-ea19822800fa-kube-api-access-r5khx\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:24:26.996191 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.996136    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12c30a4-8d8a-41aa-868a-ea19822800fa-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:24:26.996191 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.996146    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12c30a4-8d8a-41aa-868a-ea19822800fa-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:24:26.996191 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.996154    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12c30a4-8d8a-41aa-868a-ea19822800fa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:24:26.996191 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:26.996164    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12c30a4-8d8a-41aa-868a-ea19822800fa-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:24:27.286425 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:27.286332    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"
Aug 04 18:24:27.286425 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:27.286378    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r" event={"ID":"d12c30a4-8d8a-41aa-868a-ea19822800fa","Type":"ContainerDied","Data":"899104a3c9054bbe0a6afa1fb8221a3667ac47ed477161d32d7901546791db96"}
Aug 04 18:24:27.286986 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:27.286451    2566 scope.go:117] "RemoveContainer" containerID="52a7296b4e83ff3dffda055923e3780566f35b2dd903ddf5ed1e1774e011cb03"
Aug 04 18:24:27.294621 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:27.294603    2566 scope.go:117] "RemoveContainer" containerID="1b87bb44d77f265d7dc7fe7901517db7cfe56816cfcf3ca0632029a5b8eeea5f"
Aug 04 18:24:27.302558 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:27.302542    2566 scope.go:117] "RemoveContainer" containerID="36eb4c8c7e70e193ba6ca70d71b4847fdde4591b471e74dd4c639339e99e252e"
Aug 04 18:24:27.322834 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:27.322767    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"]
Aug 04 18:24:27.323815 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:27.323795    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r"]
Aug 04 18:24:27.801558 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:27.801509    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-mhm9r" podUID="d12c30a4-8d8a-41aa-868a-ea19822800fa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Aug 04 18:24:29.194511 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:29.194476    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d12c30a4-8d8a-41aa-868a-ea19822800fa" path="/var/lib/kubelet/pods/d12c30a4-8d8a-41aa-868a-ea19822800fa/volumes"
Aug 04 18:24:30.064664 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.064633    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"]
Aug 04 18:24:30.065002 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.064990    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12c30a4-8d8a-41aa-868a-ea19822800fa" containerName="kserve-container"
Aug 04 18:24:30.065051 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.065005    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12c30a4-8d8a-41aa-868a-ea19822800fa" containerName="kserve-container"
Aug 04 18:24:30.065051 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.065026    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12c30a4-8d8a-41aa-868a-ea19822800fa" containerName="kube-rbac-proxy"
Aug 04 18:24:30.065051 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.065032    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12c30a4-8d8a-41aa-868a-ea19822800fa" containerName="kube-rbac-proxy"
Aug 04 18:24:30.065051 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.065043    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12c30a4-8d8a-41aa-868a-ea19822800fa" containerName="storage-initializer"
Aug 04 18:24:30.065051 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.065049    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12c30a4-8d8a-41aa-868a-ea19822800fa" containerName="storage-initializer"
Aug 04 18:24:30.065206 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.065100    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12c30a4-8d8a-41aa-868a-ea19822800fa" containerName="kserve-container"
Aug 04 18:24:30.065206 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.065108    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12c30a4-8d8a-41aa-868a-ea19822800fa" containerName="kube-rbac-proxy"
Aug 04 18:24:30.069813 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.069795    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.079824 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.079805    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:24:30.079947 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.079811    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:24:30.080210 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.080190    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Aug 04 18:24:30.080475 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.080461    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Aug 04 18:24:30.080520 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.080508    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:24:30.081172 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.081159    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:24:30.084777 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.084761    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:24:30.105128 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.105099    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"]
Aug 04 18:24:30.122129 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.122105    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh4w4\" (UniqueName: \"kubernetes.io/projected/1f498eb4-c7c3-4448-83d0-becc2057f4af-kube-api-access-xh4w4\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.122254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.122161    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f498eb4-c7c3-4448-83d0-becc2057f4af-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.122254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.122189    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f498eb4-c7c3-4448-83d0-becc2057f4af-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.122254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.122217    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f498eb4-c7c3-4448-83d0-becc2057f4af-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.122254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.122243    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f498eb4-c7c3-4448-83d0-becc2057f4af-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.223097 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.223063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f498eb4-c7c3-4448-83d0-becc2057f4af-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.223562 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.223125    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh4w4\" (UniqueName: \"kubernetes.io/projected/1f498eb4-c7c3-4448-83d0-becc2057f4af-kube-api-access-xh4w4\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.223562 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.223174    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f498eb4-c7c3-4448-83d0-becc2057f4af-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.223562 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.223200    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f498eb4-c7c3-4448-83d0-becc2057f4af-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.223562 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.223240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f498eb4-c7c3-4448-83d0-becc2057f4af-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.223562 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:24:30.223341    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Aug 04 18:24:30.223562 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:24:30.223469    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f498eb4-c7c3-4448-83d0-becc2057f4af-proxy-tls podName:1f498eb4-c7c3-4448-83d0-becc2057f4af nodeName:}" failed. No retries permitted until 2026-08-04 18:24:30.723447388 +0000 UTC m=+1470.014061537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1f498eb4-c7c3-4448-83d0-becc2057f4af-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4" (UID: "1f498eb4-c7c3-4448-83d0-becc2057f4af") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Aug 04 18:24:30.223562 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.223534    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f498eb4-c7c3-4448-83d0-becc2057f4af-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.223841 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.223818    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f498eb4-c7c3-4448-83d0-becc2057f4af-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.223880 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.223825    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f498eb4-c7c3-4448-83d0-becc2057f4af-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.235309 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.235284    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh4w4\" (UniqueName: \"kubernetes.io/projected/1f498eb4-c7c3-4448-83d0-becc2057f4af-kube-api-access-xh4w4\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.726780 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.726742    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f498eb4-c7c3-4448-83d0-becc2057f4af-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.729164 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.729130    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f498eb4-c7c3-4448-83d0-becc2057f4af-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:30.979835 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:30.979734    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:31.147987 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:31.147959    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"]
Aug 04 18:24:31.150590 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:24:31.150552    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f498eb4_c7c3_4448_83d0_becc2057f4af.slice/crio-4ac56b526c1317ac5dbf15f2d202cb9447578dfc695df267c56da0c2c1061b25 WatchSource:0}: Error finding container 4ac56b526c1317ac5dbf15f2d202cb9447578dfc695df267c56da0c2c1061b25: Status 404 returned error can't find the container with id 4ac56b526c1317ac5dbf15f2d202cb9447578dfc695df267c56da0c2c1061b25
Aug 04 18:24:31.308094 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:31.308059    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4" event={"ID":"1f498eb4-c7c3-4448-83d0-becc2057f4af","Type":"ContainerStarted","Data":"3f1827c6c31ced4c089f7587390f88489b9fac831f3f8156873be530551c55c6"}
Aug 04 18:24:31.308604 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:31.308118    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4" event={"ID":"1f498eb4-c7c3-4448-83d0-becc2057f4af","Type":"ContainerStarted","Data":"4ac56b526c1317ac5dbf15f2d202cb9447578dfc695df267c56da0c2c1061b25"}
Aug 04 18:24:36.337908 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:36.337875    2566 generic.go:356] "Generic (PLEG): container finished" podID="1f498eb4-c7c3-4448-83d0-becc2057f4af" containerID="3f1827c6c31ced4c089f7587390f88489b9fac831f3f8156873be530551c55c6" exitCode=0
Aug 04 18:24:36.338277 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:36.337947    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4" event={"ID":"1f498eb4-c7c3-4448-83d0-becc2057f4af","Type":"ContainerDied","Data":"3f1827c6c31ced4c089f7587390f88489b9fac831f3f8156873be530551c55c6"}
Aug 04 18:24:37.346476 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:37.346445    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4" event={"ID":"1f498eb4-c7c3-4448-83d0-becc2057f4af","Type":"ContainerStarted","Data":"190823af2748f9b1fc506a7cc66eb8a31d2e24d4e4ddd129d15d89cd44565f58"}
Aug 04 18:24:37.346882 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:37.346504    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4" event={"ID":"1f498eb4-c7c3-4448-83d0-becc2057f4af","Type":"ContainerStarted","Data":"f89b785edc9831a7ac4de3e7eb8c981f1c19c66b8726f09c7eecf76c584bf8cf"}
Aug 04 18:24:37.346882 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:37.346838    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:37.346971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:37.346926    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:24:37.375478 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:37.375417    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4" podStartSLOduration=2.6058318 podStartE2EDuration="7.37539969s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.76956789s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:24:37.372972721 +0000 UTC m=+1476.663586915" watchObservedRunningTime="2026-08-04 18:24:37.37539969 +0000 UTC m=+1476.666013861"
Aug 04 18:24:43.362390 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:24:43.362339    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:26:08.193272 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:08.193238    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:26:16.517858 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:16.517816    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"]
Aug 04 18:26:16.518350 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:16.518161    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4" podUID="1f498eb4-c7c3-4448-83d0-becc2057f4af" containerName="kserve-container" containerID="cri-o://f89b785edc9831a7ac4de3e7eb8c981f1c19c66b8726f09c7eecf76c584bf8cf" gracePeriod=30
Aug 04 18:26:16.518350 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:16.518192    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4" podUID="1f498eb4-c7c3-4448-83d0-becc2057f4af" containerName="kube-rbac-proxy" containerID="cri-o://190823af2748f9b1fc506a7cc66eb8a31d2e24d4e4ddd129d15d89cd44565f58" gracePeriod=30
Aug 04 18:26:16.796702 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:16.796617    2566 generic.go:356] "Generic (PLEG): container finished" podID="1f498eb4-c7c3-4448-83d0-becc2057f4af" containerID="190823af2748f9b1fc506a7cc66eb8a31d2e24d4e4ddd129d15d89cd44565f58" exitCode=2
Aug 04 18:26:16.796702 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:16.796653    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4" event={"ID":"1f498eb4-c7c3-4448-83d0-becc2057f4af","Type":"ContainerDied","Data":"190823af2748f9b1fc506a7cc66eb8a31d2e24d4e4ddd129d15d89cd44565f58"}
Aug 04 18:26:17.975214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:17.975189    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:26:18.100413 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.100282    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f498eb4-c7c3-4448-83d0-becc2057f4af-kserve-provision-location\") pod \"1f498eb4-c7c3-4448-83d0-becc2057f4af\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") "
Aug 04 18:26:18.100413 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.100374    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f498eb4-c7c3-4448-83d0-becc2057f4af-kube-rbac-proxy-sar-config\") pod \"1f498eb4-c7c3-4448-83d0-becc2057f4af\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") "
Aug 04 18:26:18.100659 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.100457    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh4w4\" (UniqueName: \"kubernetes.io/projected/1f498eb4-c7c3-4448-83d0-becc2057f4af-kube-api-access-xh4w4\") pod \"1f498eb4-c7c3-4448-83d0-becc2057f4af\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") "
Aug 04 18:26:18.100659 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.100623    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f498eb4-c7c3-4448-83d0-becc2057f4af-cabundle-cert\") pod \"1f498eb4-c7c3-4448-83d0-becc2057f4af\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") "
Aug 04 18:26:18.100777 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.100668    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f498eb4-c7c3-4448-83d0-becc2057f4af-proxy-tls\") pod \"1f498eb4-c7c3-4448-83d0-becc2057f4af\" (UID: \"1f498eb4-c7c3-4448-83d0-becc2057f4af\") "
Aug 04 18:26:18.100777 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.100759    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f498eb4-c7c3-4448-83d0-becc2057f4af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1f498eb4-c7c3-4448-83d0-becc2057f4af" (UID: "1f498eb4-c7c3-4448-83d0-becc2057f4af"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:26:18.101018 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.100998    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f498eb4-c7c3-4448-83d0-becc2057f4af-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1f498eb4-c7c3-4448-83d0-becc2057f4af" (UID: "1f498eb4-c7c3-4448-83d0-becc2057f4af"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:26:18.101018 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.101010    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f498eb4-c7c3-4448-83d0-becc2057f4af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:26:18.102610 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.102583    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f498eb4-c7c3-4448-83d0-becc2057f4af-kube-api-access-xh4w4" (OuterVolumeSpecName: "kube-api-access-xh4w4") pod "1f498eb4-c7c3-4448-83d0-becc2057f4af" (UID: "1f498eb4-c7c3-4448-83d0-becc2057f4af"). InnerVolumeSpecName "kube-api-access-xh4w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:26:18.102711 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.102694    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f498eb4-c7c3-4448-83d0-becc2057f4af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f498eb4-c7c3-4448-83d0-becc2057f4af" (UID: "1f498eb4-c7c3-4448-83d0-becc2057f4af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:26:18.105007 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.104987    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f498eb4-c7c3-4448-83d0-becc2057f4af-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1f498eb4-c7c3-4448-83d0-becc2057f4af" (UID: "1f498eb4-c7c3-4448-83d0-becc2057f4af"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:26:18.201493 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.201459    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh4w4\" (UniqueName: \"kubernetes.io/projected/1f498eb4-c7c3-4448-83d0-becc2057f4af-kube-api-access-xh4w4\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:26:18.201493 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.201487    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f498eb4-c7c3-4448-83d0-becc2057f4af-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:26:18.201493 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.201496    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f498eb4-c7c3-4448-83d0-becc2057f4af-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:26:18.201724 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.201506    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f498eb4-c7c3-4448-83d0-becc2057f4af-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:26:18.807510 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.807475    2566 generic.go:356] "Generic (PLEG): container finished" podID="1f498eb4-c7c3-4448-83d0-becc2057f4af" containerID="f89b785edc9831a7ac4de3e7eb8c981f1c19c66b8726f09c7eecf76c584bf8cf" exitCode=0
Aug 04 18:26:18.807711 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.807540    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"
Aug 04 18:26:18.807711 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.807552    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4" event={"ID":"1f498eb4-c7c3-4448-83d0-becc2057f4af","Type":"ContainerDied","Data":"f89b785edc9831a7ac4de3e7eb8c981f1c19c66b8726f09c7eecf76c584bf8cf"}
Aug 04 18:26:18.807711 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.807615    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4" event={"ID":"1f498eb4-c7c3-4448-83d0-becc2057f4af","Type":"ContainerDied","Data":"4ac56b526c1317ac5dbf15f2d202cb9447578dfc695df267c56da0c2c1061b25"}
Aug 04 18:26:18.807711 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.807641    2566 scope.go:117] "RemoveContainer" containerID="190823af2748f9b1fc506a7cc66eb8a31d2e24d4e4ddd129d15d89cd44565f58"
Aug 04 18:26:18.816348 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.816325    2566 scope.go:117] "RemoveContainer" containerID="f89b785edc9831a7ac4de3e7eb8c981f1c19c66b8726f09c7eecf76c584bf8cf"
Aug 04 18:26:18.824245 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.824227    2566 scope.go:117] "RemoveContainer" containerID="3f1827c6c31ced4c089f7587390f88489b9fac831f3f8156873be530551c55c6"
Aug 04 18:26:18.831881 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.831865    2566 scope.go:117] "RemoveContainer" containerID="190823af2748f9b1fc506a7cc66eb8a31d2e24d4e4ddd129d15d89cd44565f58"
Aug 04 18:26:18.832139 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:26:18.832124    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190823af2748f9b1fc506a7cc66eb8a31d2e24d4e4ddd129d15d89cd44565f58\": container with ID starting with 190823af2748f9b1fc506a7cc66eb8a31d2e24d4e4ddd129d15d89cd44565f58 not found: ID does not exist" containerID="190823af2748f9b1fc506a7cc66eb8a31d2e24d4e4ddd129d15d89cd44565f58"
Aug 04 18:26:18.832178 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.832147    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190823af2748f9b1fc506a7cc66eb8a31d2e24d4e4ddd129d15d89cd44565f58"} err="failed to get container status \"190823af2748f9b1fc506a7cc66eb8a31d2e24d4e4ddd129d15d89cd44565f58\": rpc error: code = NotFound desc = could not find container \"190823af2748f9b1fc506a7cc66eb8a31d2e24d4e4ddd129d15d89cd44565f58\": container with ID starting with 190823af2748f9b1fc506a7cc66eb8a31d2e24d4e4ddd129d15d89cd44565f58 not found: ID does not exist"
Aug 04 18:26:18.832178 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.832167    2566 scope.go:117] "RemoveContainer" containerID="f89b785edc9831a7ac4de3e7eb8c981f1c19c66b8726f09c7eecf76c584bf8cf"
Aug 04 18:26:18.832417 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:26:18.832393    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89b785edc9831a7ac4de3e7eb8c981f1c19c66b8726f09c7eecf76c584bf8cf\": container with ID starting with f89b785edc9831a7ac4de3e7eb8c981f1c19c66b8726f09c7eecf76c584bf8cf not found: ID does not exist" containerID="f89b785edc9831a7ac4de3e7eb8c981f1c19c66b8726f09c7eecf76c584bf8cf"
Aug 04 18:26:18.832460 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.832425    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89b785edc9831a7ac4de3e7eb8c981f1c19c66b8726f09c7eecf76c584bf8cf"} err="failed to get container status \"f89b785edc9831a7ac4de3e7eb8c981f1c19c66b8726f09c7eecf76c584bf8cf\": rpc error: code = NotFound desc = could not find container \"f89b785edc9831a7ac4de3e7eb8c981f1c19c66b8726f09c7eecf76c584bf8cf\": container with ID starting with f89b785edc9831a7ac4de3e7eb8c981f1c19c66b8726f09c7eecf76c584bf8cf not found: ID does not exist"
Aug 04 18:26:18.832460 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.832444    2566 scope.go:117] "RemoveContainer" containerID="3f1827c6c31ced4c089f7587390f88489b9fac831f3f8156873be530551c55c6"
Aug 04 18:26:18.832687 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:26:18.832670    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f1827c6c31ced4c089f7587390f88489b9fac831f3f8156873be530551c55c6\": container with ID starting with 3f1827c6c31ced4c089f7587390f88489b9fac831f3f8156873be530551c55c6 not found: ID does not exist" containerID="3f1827c6c31ced4c089f7587390f88489b9fac831f3f8156873be530551c55c6"
Aug 04 18:26:18.832723 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.832693    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f1827c6c31ced4c089f7587390f88489b9fac831f3f8156873be530551c55c6"} err="failed to get container status \"3f1827c6c31ced4c089f7587390f88489b9fac831f3f8156873be530551c55c6\": rpc error: code = NotFound desc = could not find container \"3f1827c6c31ced4c089f7587390f88489b9fac831f3f8156873be530551c55c6\": container with ID starting with 3f1827c6c31ced4c089f7587390f88489b9fac831f3f8156873be530551c55c6 not found: ID does not exist"
Aug 04 18:26:18.839785 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.839758    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"]
Aug 04 18:26:18.851250 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:18.851221    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-7nlw4"]
Aug 04 18:26:19.193989 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:19.193916    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f498eb4-c7c3-4448-83d0-becc2057f4af" path="/var/lib/kubelet/pods/1f498eb4-c7c3-4448-83d0-becc2057f4af/volumes"
Aug 04 18:26:21.614651 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.614618    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"]
Aug 04 18:26:21.615018 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.614954    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f498eb4-c7c3-4448-83d0-becc2057f4af" containerName="kserve-container"
Aug 04 18:26:21.615018 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.614965    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f498eb4-c7c3-4448-83d0-becc2057f4af" containerName="kserve-container"
Aug 04 18:26:21.615018 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.614980    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f498eb4-c7c3-4448-83d0-becc2057f4af" containerName="kube-rbac-proxy"
Aug 04 18:26:21.615018 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.614986    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f498eb4-c7c3-4448-83d0-becc2057f4af" containerName="kube-rbac-proxy"
Aug 04 18:26:21.615018 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.614998    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f498eb4-c7c3-4448-83d0-becc2057f4af" containerName="storage-initializer"
Aug 04 18:26:21.615018 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.615005    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f498eb4-c7c3-4448-83d0-becc2057f4af" containerName="storage-initializer"
Aug 04 18:26:21.615204 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.615061    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f498eb4-c7c3-4448-83d0-becc2057f4af" containerName="kserve-container"
Aug 04 18:26:21.615204 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.615068    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f498eb4-c7c3-4448-83d0-becc2057f4af" containerName="kube-rbac-proxy"
Aug 04 18:26:21.620348 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.620320    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:21.624101 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.624083    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:26:21.624284 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.624269    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:26:21.624465 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.624447    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Aug 04 18:26:21.624601 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.624581    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:26:21.624991 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.624974    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Aug 04 18:26:21.625077 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.624993    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:26:21.625077 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.624999    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:26:21.631743 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.631722    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"]
Aug 04 18:26:21.735678 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.735644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8f25866-8e85-4170-aab8-f3c583dc03ac-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:21.735849 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.735718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8f25866-8e85-4170-aab8-f3c583dc03ac-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:21.735849 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.735771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zddj\" (UniqueName: \"kubernetes.io/projected/b8f25866-8e85-4170-aab8-f3c583dc03ac-kube-api-access-7zddj\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:21.735929 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.735841    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8f25866-8e85-4170-aab8-f3c583dc03ac-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:21.735929 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.735898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8f25866-8e85-4170-aab8-f3c583dc03ac-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:21.836320 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.836285    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8f25866-8e85-4170-aab8-f3c583dc03ac-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:21.836523 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.836363    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zddj\" (UniqueName: \"kubernetes.io/projected/b8f25866-8e85-4170-aab8-f3c583dc03ac-kube-api-access-7zddj\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:21.836523 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.836388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8f25866-8e85-4170-aab8-f3c583dc03ac-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:21.836523 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.836421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8f25866-8e85-4170-aab8-f3c583dc03ac-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:21.836523 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.836443    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8f25866-8e85-4170-aab8-f3c583dc03ac-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:21.836900 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.836869    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8f25866-8e85-4170-aab8-f3c583dc03ac-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:21.837101 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.837081    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8f25866-8e85-4170-aab8-f3c583dc03ac-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:21.837172 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.837159    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8f25866-8e85-4170-aab8-f3c583dc03ac-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:21.838924 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.838909    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8f25866-8e85-4170-aab8-f3c583dc03ac-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:21.850718 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.850686    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zddj\" (UniqueName: \"kubernetes.io/projected/b8f25866-8e85-4170-aab8-f3c583dc03ac-kube-api-access-7zddj\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:21.932196 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:21.932105    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:22.064928 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:22.064847    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"]
Aug 04 18:26:22.067248 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:26:22.067217    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f25866_8e85_4170_aab8_f3c583dc03ac.slice/crio-aa63c39abdc2d694cec7c7cf15dd667fe17ea7effb54f88f21d5d02ce2a3a26e WatchSource:0}: Error finding container aa63c39abdc2d694cec7c7cf15dd667fe17ea7effb54f88f21d5d02ce2a3a26e: Status 404 returned error can't find the container with id aa63c39abdc2d694cec7c7cf15dd667fe17ea7effb54f88f21d5d02ce2a3a26e
Aug 04 18:26:22.828144 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:22.828104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt" event={"ID":"b8f25866-8e85-4170-aab8-f3c583dc03ac","Type":"ContainerStarted","Data":"c2e1250717895a559fa501eeaace7bb7423e815fa3c2e3d16c90444a5c44eeea"}
Aug 04 18:26:22.828648 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:22.828167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt" event={"ID":"b8f25866-8e85-4170-aab8-f3c583dc03ac","Type":"ContainerStarted","Data":"aa63c39abdc2d694cec7c7cf15dd667fe17ea7effb54f88f21d5d02ce2a3a26e"}
Aug 04 18:26:23.015444 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.015411    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"]
Aug 04 18:26:23.019049 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.019031    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.021535 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.021505    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Aug 04 18:26:23.021649 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.021541    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Aug 04 18:26:23.030865 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.030840    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"]
Aug 04 18:26:23.148908 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.148825    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-proxy-tls\") pod \"isvc-sklearn-predictor-857f89cbc5-thgds\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.148908 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.148878    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-857f89cbc5-thgds\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.149105 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.148958    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kserve-provision-location\") pod \"isvc-sklearn-predictor-857f89cbc5-thgds\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.149105 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.148990    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjlfx\" (UniqueName: \"kubernetes.io/projected/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kube-api-access-rjlfx\") pod \"isvc-sklearn-predictor-857f89cbc5-thgds\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.149105 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.149015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-cabundle-cert\") pod \"isvc-sklearn-predictor-857f89cbc5-thgds\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.249688 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.249657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-proxy-tls\") pod \"isvc-sklearn-predictor-857f89cbc5-thgds\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.249891 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.249704    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-857f89cbc5-thgds\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.249891 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.249734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kserve-provision-location\") pod \"isvc-sklearn-predictor-857f89cbc5-thgds\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.249891 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.249755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjlfx\" (UniqueName: \"kubernetes.io/projected/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kube-api-access-rjlfx\") pod \"isvc-sklearn-predictor-857f89cbc5-thgds\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.249891 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.249774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-cabundle-cert\") pod \"isvc-sklearn-predictor-857f89cbc5-thgds\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.250290 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.250256    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kserve-provision-location\") pod \"isvc-sklearn-predictor-857f89cbc5-thgds\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.250500 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.250481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-cabundle-cert\") pod \"isvc-sklearn-predictor-857f89cbc5-thgds\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.250625 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.250606    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-857f89cbc5-thgds\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.252251 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.252233    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-proxy-tls\") pod \"isvc-sklearn-predictor-857f89cbc5-thgds\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.264800 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.264776    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjlfx\" (UniqueName: \"kubernetes.io/projected/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kube-api-access-rjlfx\") pod \"isvc-sklearn-predictor-857f89cbc5-thgds\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.329727 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.329691    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:23.460002 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.459979    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"]
Aug 04 18:26:23.461984 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:26:23.461954    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3c9668_bb2b_44bd_ad8b_fb936ae11279.slice/crio-c3c59da12469cb798c86d0533789fc068dfb252a243694432aa9ce945b93a9f2 WatchSource:0}: Error finding container c3c59da12469cb798c86d0533789fc068dfb252a243694432aa9ce945b93a9f2: Status 404 returned error can't find the container with id c3c59da12469cb798c86d0533789fc068dfb252a243694432aa9ce945b93a9f2
Aug 04 18:26:23.835055 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.835019    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" event={"ID":"2c3c9668-bb2b-44bd-ad8b-fb936ae11279","Type":"ContainerStarted","Data":"4f8c8bd5321ded0b53ea15f4d3cec8e79634b9ee8bdb1f64c3ea462026d30ad7"}
Aug 04 18:26:23.835494 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:23.835096    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" event={"ID":"2c3c9668-bb2b-44bd-ad8b-fb936ae11279","Type":"ContainerStarted","Data":"c3c59da12469cb798c86d0533789fc068dfb252a243694432aa9ce945b93a9f2"}
Aug 04 18:26:27.857174 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:27.857139    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerID="4f8c8bd5321ded0b53ea15f4d3cec8e79634b9ee8bdb1f64c3ea462026d30ad7" exitCode=0
Aug 04 18:26:27.857587 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:27.857206    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" event={"ID":"2c3c9668-bb2b-44bd-ad8b-fb936ae11279","Type":"ContainerDied","Data":"4f8c8bd5321ded0b53ea15f4d3cec8e79634b9ee8bdb1f64c3ea462026d30ad7"}
Aug 04 18:26:28.864390 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:28.864348    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" event={"ID":"2c3c9668-bb2b-44bd-ad8b-fb936ae11279","Type":"ContainerStarted","Data":"1ab4116241351c877a2eb55ca8ca52b81aad921ebe9cf566166e02d63df6f108"}
Aug 04 18:26:28.864822 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:28.864421    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" event={"ID":"2c3c9668-bb2b-44bd-ad8b-fb936ae11279","Type":"ContainerStarted","Data":"a8dd286ccd97366a03f83d70bcf002924c7495046383778e9f611a7aabbf2043"}
Aug 04 18:26:28.864822 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:28.864709    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:28.864948 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:28.864861    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:28.866031 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:28.866006    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Aug 04 18:26:28.894264 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:28.894220    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" podStartSLOduration=3.147176956 podStartE2EDuration="6.894205152s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.747028196s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:26:28.893479054 +0000 UTC m=+1588.184093227" watchObservedRunningTime="2026-08-04 18:26:28.894205152 +0000 UTC m=+1588.184819323"
Aug 04 18:26:29.869421 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:29.869383    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Aug 04 18:26:30.875348 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:30.875317    2566 generic.go:356] "Generic (PLEG): container finished" podID="b8f25866-8e85-4170-aab8-f3c583dc03ac" containerID="c2e1250717895a559fa501eeaace7bb7423e815fa3c2e3d16c90444a5c44eeea" exitCode=0
Aug 04 18:26:30.875827 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:30.875397    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt" event={"ID":"b8f25866-8e85-4170-aab8-f3c583dc03ac","Type":"ContainerDied","Data":"c2e1250717895a559fa501eeaace7bb7423e815fa3c2e3d16c90444a5c44eeea"}
Aug 04 18:26:31.882429 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:31.882397    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt" event={"ID":"b8f25866-8e85-4170-aab8-f3c583dc03ac","Type":"ContainerStarted","Data":"f51dacf0d02da0aea674da7a89cc438f7b596e1c726bdf1fdc3063db8e4fe130"}
Aug 04 18:26:31.882890 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:31.882460    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt" event={"ID":"b8f25866-8e85-4170-aab8-f3c583dc03ac","Type":"ContainerStarted","Data":"5d1e31fc623e8645869d41fd7a61e89991d6b49eb7ca7de96d1016b45b2db2c8"}
Aug 04 18:26:31.882890 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:31.882669    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:31.904057 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:31.904000    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt" podStartSLOduration=2.19799833 podStartE2EDuration="10.903983995s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.705985665s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:26:31.90317606 +0000 UTC m=+1591.193790256" watchObservedRunningTime="2026-08-04 18:26:31.903983995 +0000 UTC m=+1591.194598166"
Aug 04 18:26:32.890795 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:32.890755    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:32.892253 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:32.892224    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt" podUID="b8f25866-8e85-4170-aab8-f3c583dc03ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Aug 04 18:26:33.895828 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:33.895785    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt" podUID="b8f25866-8e85-4170-aab8-f3c583dc03ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Aug 04 18:26:34.874265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:34.874230    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:26:34.874861 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:34.874830    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Aug 04 18:26:38.900708 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:38.900682    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:38.901224 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:38.901190    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt" podUID="b8f25866-8e85-4170-aab8-f3c583dc03ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Aug 04 18:26:44.875236 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:44.875192    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Aug 04 18:26:48.902228 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:48.902193    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:26:54.875186 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:26:54.875088    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Aug 04 18:27:04.214432 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:04.214401    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"]
Aug 04 18:27:04.214864 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:04.214641    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt" podUID="b8f25866-8e85-4170-aab8-f3c583dc03ac" containerName="kserve-container" containerID="cri-o://5d1e31fc623e8645869d41fd7a61e89991d6b49eb7ca7de96d1016b45b2db2c8" gracePeriod=30
Aug 04 18:27:04.214864 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:04.214693    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt" podUID="b8f25866-8e85-4170-aab8-f3c583dc03ac" containerName="kube-rbac-proxy" containerID="cri-o://f51dacf0d02da0aea674da7a89cc438f7b596e1c726bdf1fdc3063db8e4fe130" gracePeriod=30
Aug 04 18:27:04.875489 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:04.875445    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Aug 04 18:27:05.050898 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:05.050864    2566 generic.go:356] "Generic (PLEG): container finished" podID="b8f25866-8e85-4170-aab8-f3c583dc03ac" containerID="f51dacf0d02da0aea674da7a89cc438f7b596e1c726bdf1fdc3063db8e4fe130" exitCode=2
Aug 04 18:27:05.051065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:05.050949    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt" event={"ID":"b8f25866-8e85-4170-aab8-f3c583dc03ac","Type":"ContainerDied","Data":"f51dacf0d02da0aea674da7a89cc438f7b596e1c726bdf1fdc3063db8e4fe130"}
Aug 04 18:27:05.958909 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:05.958885    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:27:06.056578 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.056549    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8f25866-8e85-4170-aab8-f3c583dc03ac-cabundle-cert\") pod \"b8f25866-8e85-4170-aab8-f3c583dc03ac\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") "
Aug 04 18:27:06.056786 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.056608    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zddj\" (UniqueName: \"kubernetes.io/projected/b8f25866-8e85-4170-aab8-f3c583dc03ac-kube-api-access-7zddj\") pod \"b8f25866-8e85-4170-aab8-f3c583dc03ac\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") "
Aug 04 18:27:06.056786 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.056649    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8f25866-8e85-4170-aab8-f3c583dc03ac-kserve-provision-location\") pod \"b8f25866-8e85-4170-aab8-f3c583dc03ac\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") "
Aug 04 18:27:06.056786 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.056676    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8f25866-8e85-4170-aab8-f3c583dc03ac-kube-rbac-proxy-sar-config\") pod \"b8f25866-8e85-4170-aab8-f3c583dc03ac\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") "
Aug 04 18:27:06.056786 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.056724    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8f25866-8e85-4170-aab8-f3c583dc03ac-proxy-tls\") pod \"b8f25866-8e85-4170-aab8-f3c583dc03ac\" (UID: \"b8f25866-8e85-4170-aab8-f3c583dc03ac\") "
Aug 04 18:27:06.057031 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.056914    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f25866-8e85-4170-aab8-f3c583dc03ac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b8f25866-8e85-4170-aab8-f3c583dc03ac" (UID: "b8f25866-8e85-4170-aab8-f3c583dc03ac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:27:06.057125 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.057093    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f25866-8e85-4170-aab8-f3c583dc03ac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b8f25866-8e85-4170-aab8-f3c583dc03ac" (UID: "b8f25866-8e85-4170-aab8-f3c583dc03ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:27:06.057818 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.057789    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f25866-8e85-4170-aab8-f3c583dc03ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8f25866-8e85-4170-aab8-f3c583dc03ac" (UID: "b8f25866-8e85-4170-aab8-f3c583dc03ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:06.058726 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.058698    2566 generic.go:356] "Generic (PLEG): container finished" podID="b8f25866-8e85-4170-aab8-f3c583dc03ac" containerID="5d1e31fc623e8645869d41fd7a61e89991d6b49eb7ca7de96d1016b45b2db2c8" exitCode=0
Aug 04 18:27:06.058842 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.058741    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt" event={"ID":"b8f25866-8e85-4170-aab8-f3c583dc03ac","Type":"ContainerDied","Data":"5d1e31fc623e8645869d41fd7a61e89991d6b49eb7ca7de96d1016b45b2db2c8"}
Aug 04 18:27:06.058842 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.058785    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"
Aug 04 18:27:06.058842 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.058807    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt" event={"ID":"b8f25866-8e85-4170-aab8-f3c583dc03ac","Type":"ContainerDied","Data":"aa63c39abdc2d694cec7c7cf15dd667fe17ea7effb54f88f21d5d02ce2a3a26e"}
Aug 04 18:27:06.058972 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.058847    2566 scope.go:117] "RemoveContainer" containerID="f51dacf0d02da0aea674da7a89cc438f7b596e1c726bdf1fdc3063db8e4fe130"
Aug 04 18:27:06.059029 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.059004    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f25866-8e85-4170-aab8-f3c583dc03ac-kube-api-access-7zddj" (OuterVolumeSpecName: "kube-api-access-7zddj") pod "b8f25866-8e85-4170-aab8-f3c583dc03ac" (UID: "b8f25866-8e85-4170-aab8-f3c583dc03ac"). InnerVolumeSpecName "kube-api-access-7zddj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:27:06.059139 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.059115    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8f25866-8e85-4170-aab8-f3c583dc03ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8f25866-8e85-4170-aab8-f3c583dc03ac" (UID: "b8f25866-8e85-4170-aab8-f3c583dc03ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:27:06.077308 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.077288    2566 scope.go:117] "RemoveContainer" containerID="5d1e31fc623e8645869d41fd7a61e89991d6b49eb7ca7de96d1016b45b2db2c8"
Aug 04 18:27:06.084770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.084751    2566 scope.go:117] "RemoveContainer" containerID="c2e1250717895a559fa501eeaace7bb7423e815fa3c2e3d16c90444a5c44eeea"
Aug 04 18:27:06.091983 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.091963    2566 scope.go:117] "RemoveContainer" containerID="f51dacf0d02da0aea674da7a89cc438f7b596e1c726bdf1fdc3063db8e4fe130"
Aug 04 18:27:06.092241 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:27:06.092222    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51dacf0d02da0aea674da7a89cc438f7b596e1c726bdf1fdc3063db8e4fe130\": container with ID starting with f51dacf0d02da0aea674da7a89cc438f7b596e1c726bdf1fdc3063db8e4fe130 not found: ID does not exist" containerID="f51dacf0d02da0aea674da7a89cc438f7b596e1c726bdf1fdc3063db8e4fe130"
Aug 04 18:27:06.092303 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.092249    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51dacf0d02da0aea674da7a89cc438f7b596e1c726bdf1fdc3063db8e4fe130"} err="failed to get container status \"f51dacf0d02da0aea674da7a89cc438f7b596e1c726bdf1fdc3063db8e4fe130\": rpc error: code = NotFound desc = could not find container \"f51dacf0d02da0aea674da7a89cc438f7b596e1c726bdf1fdc3063db8e4fe130\": container with ID starting with f51dacf0d02da0aea674da7a89cc438f7b596e1c726bdf1fdc3063db8e4fe130 not found: ID does not exist"
Aug 04 18:27:06.092303 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.092267    2566 scope.go:117] "RemoveContainer" containerID="5d1e31fc623e8645869d41fd7a61e89991d6b49eb7ca7de96d1016b45b2db2c8"
Aug 04 18:27:06.092530 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:27:06.092516    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1e31fc623e8645869d41fd7a61e89991d6b49eb7ca7de96d1016b45b2db2c8\": container with ID starting with 5d1e31fc623e8645869d41fd7a61e89991d6b49eb7ca7de96d1016b45b2db2c8 not found: ID does not exist" containerID="5d1e31fc623e8645869d41fd7a61e89991d6b49eb7ca7de96d1016b45b2db2c8"
Aug 04 18:27:06.092576 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.092535    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1e31fc623e8645869d41fd7a61e89991d6b49eb7ca7de96d1016b45b2db2c8"} err="failed to get container status \"5d1e31fc623e8645869d41fd7a61e89991d6b49eb7ca7de96d1016b45b2db2c8\": rpc error: code = NotFound desc = could not find container \"5d1e31fc623e8645869d41fd7a61e89991d6b49eb7ca7de96d1016b45b2db2c8\": container with ID starting with 5d1e31fc623e8645869d41fd7a61e89991d6b49eb7ca7de96d1016b45b2db2c8 not found: ID does not exist"
Aug 04 18:27:06.092576 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.092546    2566 scope.go:117] "RemoveContainer" containerID="c2e1250717895a559fa501eeaace7bb7423e815fa3c2e3d16c90444a5c44eeea"
Aug 04 18:27:06.092761 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:27:06.092746    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e1250717895a559fa501eeaace7bb7423e815fa3c2e3d16c90444a5c44eeea\": container with ID starting with c2e1250717895a559fa501eeaace7bb7423e815fa3c2e3d16c90444a5c44eeea not found: ID does not exist" containerID="c2e1250717895a559fa501eeaace7bb7423e815fa3c2e3d16c90444a5c44eeea"
Aug 04 18:27:06.092799 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.092767    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e1250717895a559fa501eeaace7bb7423e815fa3c2e3d16c90444a5c44eeea"} err="failed to get container status \"c2e1250717895a559fa501eeaace7bb7423e815fa3c2e3d16c90444a5c44eeea\": rpc error: code = NotFound desc = could not find container \"c2e1250717895a559fa501eeaace7bb7423e815fa3c2e3d16c90444a5c44eeea\": container with ID starting with c2e1250717895a559fa501eeaace7bb7423e815fa3c2e3d16c90444a5c44eeea not found: ID does not exist"
Aug 04 18:27:06.158172 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.158139    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8f25866-8e85-4170-aab8-f3c583dc03ac-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:27:06.158403 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.158385    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zddj\" (UniqueName: \"kubernetes.io/projected/b8f25866-8e85-4170-aab8-f3c583dc03ac-kube-api-access-7zddj\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:27:06.158507 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.158492    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8f25866-8e85-4170-aab8-f3c583dc03ac-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:27:06.158598 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.158587    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8f25866-8e85-4170-aab8-f3c583dc03ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:27:06.158672 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.158662    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8f25866-8e85-4170-aab8-f3c583dc03ac-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:27:06.383861 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.383828    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"]
Aug 04 18:27:06.388792 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:06.388767    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9b7zt"]
Aug 04 18:27:07.194864 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:07.194832    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8f25866-8e85-4170-aab8-f3c583dc03ac" path="/var/lib/kubelet/pods/b8f25866-8e85-4170-aab8-f3c583dc03ac/volumes"
Aug 04 18:27:14.875099 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:14.875057    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Aug 04 18:27:24.875597 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:24.875560    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Aug 04 18:27:34.875506 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:34.875465    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Aug 04 18:27:44.875303 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:44.875265    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:27:53.230152 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:53.230116    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"]
Aug 04 18:27:53.230620 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:53.230388    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kserve-container" containerID="cri-o://a8dd286ccd97366a03f83d70bcf002924c7495046383778e9f611a7aabbf2043" gracePeriod=30
Aug 04 18:27:53.230620 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:53.230424    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kube-rbac-proxy" containerID="cri-o://1ab4116241351c877a2eb55ca8ca52b81aad921ebe9cf566166e02d63df6f108" gracePeriod=30
Aug 04 18:27:54.291992 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:54.291958    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerID="1ab4116241351c877a2eb55ca8ca52b81aad921ebe9cf566166e02d63df6f108" exitCode=2
Aug 04 18:27:54.292391 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:54.292031    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" event={"ID":"2c3c9668-bb2b-44bd-ad8b-fb936ae11279","Type":"ContainerDied","Data":"1ab4116241351c877a2eb55ca8ca52b81aad921ebe9cf566166e02d63df6f108"}
Aug 04 18:27:54.869777 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:54.869739    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Aug 04 18:27:54.875156 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:54.875128    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Aug 04 18:27:57.967465 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:57.967437    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:27:58.032682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.032608    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjlfx\" (UniqueName: \"kubernetes.io/projected/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kube-api-access-rjlfx\") pod \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") "
Aug 04 18:27:58.032682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.032651    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kube-rbac-proxy-sar-config\") pod \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") "
Aug 04 18:27:58.032682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.032672    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-proxy-tls\") pod \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") "
Aug 04 18:27:58.032915 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.032761    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-cabundle-cert\") pod \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") "
Aug 04 18:27:58.032915 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.032801    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kserve-provision-location\") pod \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\" (UID: \"2c3c9668-bb2b-44bd-ad8b-fb936ae11279\") "
Aug 04 18:27:58.033042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.033014    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c3c9668-bb2b-44bd-ad8b-fb936ae11279" (UID: "2c3c9668-bb2b-44bd-ad8b-fb936ae11279"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:27:58.033153 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.033119    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c3c9668-bb2b-44bd-ad8b-fb936ae11279" (UID: "2c3c9668-bb2b-44bd-ad8b-fb936ae11279"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:27:58.033290 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.033198    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c3c9668-bb2b-44bd-ad8b-fb936ae11279" (UID: "2c3c9668-bb2b-44bd-ad8b-fb936ae11279"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:58.034780 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.034761    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kube-api-access-rjlfx" (OuterVolumeSpecName: "kube-api-access-rjlfx") pod "2c3c9668-bb2b-44bd-ad8b-fb936ae11279" (UID: "2c3c9668-bb2b-44bd-ad8b-fb936ae11279"). InnerVolumeSpecName "kube-api-access-rjlfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:27:58.034874 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.034773    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c3c9668-bb2b-44bd-ad8b-fb936ae11279" (UID: "2c3c9668-bb2b-44bd-ad8b-fb936ae11279"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:27:58.134426 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.134393    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:27:58.134426 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.134418    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:27:58.134426 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.134428    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjlfx\" (UniqueName: \"kubernetes.io/projected/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kube-api-access-rjlfx\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:27:58.134651 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.134437    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:27:58.134651 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.134448    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c3c9668-bb2b-44bd-ad8b-fb936ae11279-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:27:58.313674 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.313636    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerID="a8dd286ccd97366a03f83d70bcf002924c7495046383778e9f611a7aabbf2043" exitCode=0
Aug 04 18:27:58.313836 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.313710    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" event={"ID":"2c3c9668-bb2b-44bd-ad8b-fb936ae11279","Type":"ContainerDied","Data":"a8dd286ccd97366a03f83d70bcf002924c7495046383778e9f611a7aabbf2043"}
Aug 04 18:27:58.313836 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.313719    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"
Aug 04 18:27:58.313836 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.313760    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds" event={"ID":"2c3c9668-bb2b-44bd-ad8b-fb936ae11279","Type":"ContainerDied","Data":"c3c59da12469cb798c86d0533789fc068dfb252a243694432aa9ce945b93a9f2"}
Aug 04 18:27:58.313836 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.313790    2566 scope.go:117] "RemoveContainer" containerID="1ab4116241351c877a2eb55ca8ca52b81aad921ebe9cf566166e02d63df6f108"
Aug 04 18:27:58.315085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315056    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"]
Aug 04 18:27:58.315592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315576    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kserve-container"
Aug 04 18:27:58.315592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315596    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kserve-container"
Aug 04 18:27:58.315707 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315609    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8f25866-8e85-4170-aab8-f3c583dc03ac" containerName="kserve-container"
Aug 04 18:27:58.315707 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315619    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8f25866-8e85-4170-aab8-f3c583dc03ac" containerName="kserve-container"
Aug 04 18:27:58.315707 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315634    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8f25866-8e85-4170-aab8-f3c583dc03ac" containerName="kube-rbac-proxy"
Aug 04 18:27:58.315707 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315643    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8f25866-8e85-4170-aab8-f3c583dc03ac" containerName="kube-rbac-proxy"
Aug 04 18:27:58.315707 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315662    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kube-rbac-proxy"
Aug 04 18:27:58.315707 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315671    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kube-rbac-proxy"
Aug 04 18:27:58.315707 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315700    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="storage-initializer"
Aug 04 18:27:58.315707 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315709    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="storage-initializer"
Aug 04 18:27:58.315976 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315726    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8f25866-8e85-4170-aab8-f3c583dc03ac" containerName="storage-initializer"
Aug 04 18:27:58.315976 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315735    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8f25866-8e85-4170-aab8-f3c583dc03ac" containerName="storage-initializer"
Aug 04 18:27:58.315976 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315817    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kube-rbac-proxy"
Aug 04 18:27:58.315976 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315830    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8f25866-8e85-4170-aab8-f3c583dc03ac" containerName="kserve-container"
Aug 04 18:27:58.315976 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315840    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8f25866-8e85-4170-aab8-f3c583dc03ac" containerName="kube-rbac-proxy"
Aug 04 18:27:58.315976 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.315852    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" containerName="kserve-container"
Aug 04 18:27:58.325270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.325017    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:58.328402 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.328380    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:27:58.328785 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.328501    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:27:58.329044 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.328561    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Aug 04 18:27:58.329044 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.328624    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Aug 04 18:27:58.329227 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.328624    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:27:58.329227 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.328667    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:27:58.329227 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.328685    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:27:58.329560 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.329540    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"]
Aug 04 18:27:58.330073 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.330054    2566 scope.go:117] "RemoveContainer" containerID="a8dd286ccd97366a03f83d70bcf002924c7495046383778e9f611a7aabbf2043"
Aug 04 18:27:58.340058 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.340038    2566 scope.go:117] "RemoveContainer" containerID="4f8c8bd5321ded0b53ea15f4d3cec8e79634b9ee8bdb1f64c3ea462026d30ad7"
Aug 04 18:27:58.348411 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.348383    2566 scope.go:117] "RemoveContainer" containerID="1ab4116241351c877a2eb55ca8ca52b81aad921ebe9cf566166e02d63df6f108"
Aug 04 18:27:58.348687 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:27:58.348670    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab4116241351c877a2eb55ca8ca52b81aad921ebe9cf566166e02d63df6f108\": container with ID starting with 1ab4116241351c877a2eb55ca8ca52b81aad921ebe9cf566166e02d63df6f108 not found: ID does not exist" containerID="1ab4116241351c877a2eb55ca8ca52b81aad921ebe9cf566166e02d63df6f108"
Aug 04 18:27:58.348735 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.348695    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab4116241351c877a2eb55ca8ca52b81aad921ebe9cf566166e02d63df6f108"} err="failed to get container status \"1ab4116241351c877a2eb55ca8ca52b81aad921ebe9cf566166e02d63df6f108\": rpc error: code = NotFound desc = could not find container \"1ab4116241351c877a2eb55ca8ca52b81aad921ebe9cf566166e02d63df6f108\": container with ID starting with 1ab4116241351c877a2eb55ca8ca52b81aad921ebe9cf566166e02d63df6f108 not found: ID does not exist"
Aug 04 18:27:58.348735 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.348710    2566 scope.go:117] "RemoveContainer" containerID="a8dd286ccd97366a03f83d70bcf002924c7495046383778e9f611a7aabbf2043"
Aug 04 18:27:58.348923 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:27:58.348909    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8dd286ccd97366a03f83d70bcf002924c7495046383778e9f611a7aabbf2043\": container with ID starting with a8dd286ccd97366a03f83d70bcf002924c7495046383778e9f611a7aabbf2043 not found: ID does not exist" containerID="a8dd286ccd97366a03f83d70bcf002924c7495046383778e9f611a7aabbf2043"
Aug 04 18:27:58.348964 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.348925    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8dd286ccd97366a03f83d70bcf002924c7495046383778e9f611a7aabbf2043"} err="failed to get container status \"a8dd286ccd97366a03f83d70bcf002924c7495046383778e9f611a7aabbf2043\": rpc error: code = NotFound desc = could not find container \"a8dd286ccd97366a03f83d70bcf002924c7495046383778e9f611a7aabbf2043\": container with ID starting with a8dd286ccd97366a03f83d70bcf002924c7495046383778e9f611a7aabbf2043 not found: ID does not exist"
Aug 04 18:27:58.348964 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.348938    2566 scope.go:117] "RemoveContainer" containerID="4f8c8bd5321ded0b53ea15f4d3cec8e79634b9ee8bdb1f64c3ea462026d30ad7"
Aug 04 18:27:58.349181 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:27:58.349163    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8c8bd5321ded0b53ea15f4d3cec8e79634b9ee8bdb1f64c3ea462026d30ad7\": container with ID starting with 4f8c8bd5321ded0b53ea15f4d3cec8e79634b9ee8bdb1f64c3ea462026d30ad7 not found: ID does not exist" containerID="4f8c8bd5321ded0b53ea15f4d3cec8e79634b9ee8bdb1f64c3ea462026d30ad7"
Aug 04 18:27:58.349242 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.349191    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8c8bd5321ded0b53ea15f4d3cec8e79634b9ee8bdb1f64c3ea462026d30ad7"} err="failed to get container status \"4f8c8bd5321ded0b53ea15f4d3cec8e79634b9ee8bdb1f64c3ea462026d30ad7\": rpc error: code = NotFound desc = could not find container \"4f8c8bd5321ded0b53ea15f4d3cec8e79634b9ee8bdb1f64c3ea462026d30ad7\": container with ID starting with 4f8c8bd5321ded0b53ea15f4d3cec8e79634b9ee8bdb1f64c3ea462026d30ad7 not found: ID does not exist"
Aug 04 18:27:58.359133 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.359112    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"]
Aug 04 18:27:58.364005 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.363984    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-857f89cbc5-thgds"]
Aug 04 18:27:58.436540 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.436501    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a074d99c-684f-4964-9d30-3b688babff3d-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:58.436661 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.436563    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a074d99c-684f-4964-9d30-3b688babff3d-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:58.436661 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.436618    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a074d99c-684f-4964-9d30-3b688babff3d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:58.436752 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.436660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a074d99c-684f-4964-9d30-3b688babff3d-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:58.436752 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.436725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m69t7\" (UniqueName: \"kubernetes.io/projected/a074d99c-684f-4964-9d30-3b688babff3d-kube-api-access-m69t7\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:58.537987 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.537953    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a074d99c-684f-4964-9d30-3b688babff3d-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:58.538107 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.538017    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a074d99c-684f-4964-9d30-3b688babff3d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:58.538107 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.538040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a074d99c-684f-4964-9d30-3b688babff3d-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:58.538107 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.538062    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m69t7\" (UniqueName: \"kubernetes.io/projected/a074d99c-684f-4964-9d30-3b688babff3d-kube-api-access-m69t7\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:58.538107 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.538095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a074d99c-684f-4964-9d30-3b688babff3d-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:58.538290 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:27:58.538186    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Aug 04 18:27:58.538290 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:27:58.538285    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a074d99c-684f-4964-9d30-3b688babff3d-proxy-tls podName:a074d99c-684f-4964-9d30-3b688babff3d nodeName:}" failed. No retries permitted until 2026-08-04 18:27:59.038264361 +0000 UTC m=+1678.328878537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a074d99c-684f-4964-9d30-3b688babff3d-proxy-tls") pod "sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8" (UID: "a074d99c-684f-4964-9d30-3b688babff3d") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Aug 04 18:27:58.538632 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.538614    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a074d99c-684f-4964-9d30-3b688babff3d-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:58.538848 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.538828    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a074d99c-684f-4964-9d30-3b688babff3d-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:58.538884 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.538828    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a074d99c-684f-4964-9d30-3b688babff3d-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:58.547177 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:58.547150    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m69t7\" (UniqueName: \"kubernetes.io/projected/a074d99c-684f-4964-9d30-3b688babff3d-kube-api-access-m69t7\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:59.042663 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:59.042622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a074d99c-684f-4964-9d30-3b688babff3d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:59.045013 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:59.044983    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a074d99c-684f-4964-9d30-3b688babff3d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:59.194220 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:59.194177    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c3c9668-bb2b-44bd-ad8b-fb936ae11279" path="/var/lib/kubelet/pods/2c3c9668-bb2b-44bd-ad8b-fb936ae11279/volumes"
Aug 04 18:27:59.238460 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:59.238418    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:27:59.387794 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:27:59.387767    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"]
Aug 04 18:27:59.390027 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:27:59.389993    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda074d99c_684f_4964_9d30_3b688babff3d.slice/crio-19dc808d4c61f2a9126a0eee249511b5f7703727b20db0abaf2c6ca4f136bc85 WatchSource:0}: Error finding container 19dc808d4c61f2a9126a0eee249511b5f7703727b20db0abaf2c6ca4f136bc85: Status 404 returned error can't find the container with id 19dc808d4c61f2a9126a0eee249511b5f7703727b20db0abaf2c6ca4f136bc85
Aug 04 18:28:00.326936 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:28:00.326903    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8" event={"ID":"a074d99c-684f-4964-9d30-3b688babff3d","Type":"ContainerStarted","Data":"46ec0cdccac6b1bd67395c7c91271328e7952a0468d92a52afc5f0dc1b726406"}
Aug 04 18:28:00.327308 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:28:00.326975    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8" event={"ID":"a074d99c-684f-4964-9d30-3b688babff3d","Type":"ContainerStarted","Data":"19dc808d4c61f2a9126a0eee249511b5f7703727b20db0abaf2c6ca4f136bc85"}
Aug 04 18:28:03.341690 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:28:03.341656    2566 generic.go:356] "Generic (PLEG): container finished" podID="a074d99c-684f-4964-9d30-3b688babff3d" containerID="46ec0cdccac6b1bd67395c7c91271328e7952a0468d92a52afc5f0dc1b726406" exitCode=0
Aug 04 18:28:03.342069 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:28:03.341697    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8" event={"ID":"a074d99c-684f-4964-9d30-3b688babff3d","Type":"ContainerDied","Data":"46ec0cdccac6b1bd67395c7c91271328e7952a0468d92a52afc5f0dc1b726406"}
Aug 04 18:28:03.342912 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:28:03.342898    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:30:00.084778 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.084745    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"]
Aug 04 18:30:00.159934 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.159889    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"]
Aug 04 18:30:00.160125 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.160046    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:00.162669 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.162635    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Aug 04 18:30:00.162669 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.162638    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Aug 04 18:30:00.180167 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.180137    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"]
Aug 04 18:30:00.180397 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.180192    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"]
Aug 04 18:30:00.180397 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.180239    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.182944 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.182921    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Aug 04 18:30:00.202875 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.202842    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-cabundle-cert\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:00.203045 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.202920    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kserve-provision-location\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:00.203045 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.203008    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-proxy-tls\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:00.203045 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.203027    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:00.203201 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.203066    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgs7t\" (UniqueName: \"kubernetes.io/projected/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kube-api-access-cgs7t\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:00.304534 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.304493    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kserve-provision-location\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:00.304719 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.304560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00ea82f3-57c6-4de8-a960-37f114fb4170-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-dq7l7\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.304719 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.304588    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00ea82f3-57c6-4de8-a960-37f114fb4170-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-dq7l7\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.304719 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.304640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00ea82f3-57c6-4de8-a960-37f114fb4170-proxy-tls\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-dq7l7\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.304719 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.304711    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-proxy-tls\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:00.304957 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.304741    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:00.304957 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:30:00.304845    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Aug 04 18:30:00.304957 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.304902    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwbw7\" (UniqueName: \"kubernetes.io/projected/00ea82f3-57c6-4de8-a960-37f114fb4170-kube-api-access-gwbw7\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-dq7l7\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.304957 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:30:00.304918    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-proxy-tls podName:67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f nodeName:}" failed. No retries permitted until 2026-08-04 18:30:00.804890636 +0000 UTC m=+1800.095504786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-proxy-tls") pod "isvc-canary-create-predictor-65d8dbfcdb-xcggl" (UID: "67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f") : secret "isvc-canary-create-predictor-serving-cert" not found
Aug 04 18:30:00.304957 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.304943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kserve-provision-location\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:00.305221 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.304962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgs7t\" (UniqueName: \"kubernetes.io/projected/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kube-api-access-cgs7t\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:00.305221 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.304990    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00ea82f3-57c6-4de8-a960-37f114fb4170-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-dq7l7\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.305221 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.305031    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-cabundle-cert\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:00.305473 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.305447    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:00.305628 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.305602    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-cabundle-cert\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:00.314554 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.314498    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgs7t\" (UniqueName: \"kubernetes.io/projected/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kube-api-access-cgs7t\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:00.405931 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.405830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00ea82f3-57c6-4de8-a960-37f114fb4170-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-dq7l7\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.406113 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.405941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00ea82f3-57c6-4de8-a960-37f114fb4170-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-dq7l7\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.406113 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.405970    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00ea82f3-57c6-4de8-a960-37f114fb4170-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-dq7l7\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.406113 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.406010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00ea82f3-57c6-4de8-a960-37f114fb4170-proxy-tls\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-dq7l7\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.406113 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.406082    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwbw7\" (UniqueName: \"kubernetes.io/projected/00ea82f3-57c6-4de8-a960-37f114fb4170-kube-api-access-gwbw7\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-dq7l7\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.406345 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.406317    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00ea82f3-57c6-4de8-a960-37f114fb4170-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-dq7l7\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.406749 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.406721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00ea82f3-57c6-4de8-a960-37f114fb4170-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-dq7l7\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.406825 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.406788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00ea82f3-57c6-4de8-a960-37f114fb4170-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-dq7l7\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.409227 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.409203    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00ea82f3-57c6-4de8-a960-37f114fb4170-proxy-tls\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-dq7l7\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.416006 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.415974    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwbw7\" (UniqueName: \"kubernetes.io/projected/00ea82f3-57c6-4de8-a960-37f114fb4170-kube-api-access-gwbw7\") pod \"isvc-canary-create-v2-predictor-d6d6859cf-dq7l7\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.492620 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.492579    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:00.810157 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:00.810118    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-proxy-tls\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:00.810346 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:30:00.810299    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Aug 04 18:30:00.810441 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:30:00.810379    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-proxy-tls podName:67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f nodeName:}" failed. No retries permitted until 2026-08-04 18:30:01.810344025 +0000 UTC m=+1801.100958189 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-proxy-tls") pod "isvc-canary-create-predictor-65d8dbfcdb-xcggl" (UID: "67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f") : secret "isvc-canary-create-predictor-serving-cert" not found
Aug 04 18:30:01.821800 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:01.821749    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-proxy-tls\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:01.824788 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:01.824739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-proxy-tls\") pod \"isvc-canary-create-predictor-65d8dbfcdb-xcggl\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:01.972962 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:01.972919    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:08.959257 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:08.959218    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"]
Aug 04 18:30:08.968075 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:30:08.968043    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f5e3d2_53a1_4a3b_bd92_e4c0eaf2491f.slice/crio-5c45a83d7d366674cd8e09dff822d562d441730377916fd793d714fd048e0cd3 WatchSource:0}: Error finding container 5c45a83d7d366674cd8e09dff822d562d441730377916fd793d714fd048e0cd3: Status 404 returned error can't find the container with id 5c45a83d7d366674cd8e09dff822d562d441730377916fd793d714fd048e0cd3
Aug 04 18:30:08.975393 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:08.975333    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"]
Aug 04 18:30:09.017131 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:09.017058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" event={"ID":"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f","Type":"ContainerStarted","Data":"5c45a83d7d366674cd8e09dff822d562d441730377916fd793d714fd048e0cd3"}
Aug 04 18:30:09.021632 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:30:09.021601    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ea82f3_57c6_4de8_a960_37f114fb4170.slice/crio-4f40c34f1720b159ad440915ce691474a4eca0e81ffd8982af61ad09c4b748f8 WatchSource:0}: Error finding container 4f40c34f1720b159ad440915ce691474a4eca0e81ffd8982af61ad09c4b748f8: Status 404 returned error can't find the container with id 4f40c34f1720b159ad440915ce691474a4eca0e81ffd8982af61ad09c4b748f8
Aug 04 18:30:10.025478 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:10.025435    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" event={"ID":"00ea82f3-57c6-4de8-a960-37f114fb4170","Type":"ContainerStarted","Data":"9b9310caab6a4caefe5354cead99129721ad7efb514b7d51515ed4c8f0de926e"}
Aug 04 18:30:10.025969 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:10.025510    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" event={"ID":"00ea82f3-57c6-4de8-a960-37f114fb4170","Type":"ContainerStarted","Data":"4f40c34f1720b159ad440915ce691474a4eca0e81ffd8982af61ad09c4b748f8"}
Aug 04 18:30:10.027659 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:10.027635    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8" event={"ID":"a074d99c-684f-4964-9d30-3b688babff3d","Type":"ContainerStarted","Data":"3c47a4ccfece022f300aa26a2d6cc77dd59619cf5d4c9abf9e2c709802e7fd5f"}
Aug 04 18:30:10.027804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:10.027683    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8" event={"ID":"a074d99c-684f-4964-9d30-3b688babff3d","Type":"ContainerStarted","Data":"f1b0d9a7cf3b04255b8297fc40d708439da6eaadf504efc513b3e444c500bbfd"}
Aug 04 18:30:10.027882 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:10.027804    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:30:10.027882 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:10.027841    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:30:10.029522 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:10.029490    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" event={"ID":"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f","Type":"ContainerStarted","Data":"d364e4a108af28fee081751ee9ae0757e3c4b1424e5eff93be3816f655fa3cd4"}
Aug 04 18:30:10.076464 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:10.076410    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8" podStartSLOduration=2.400660679 podStartE2EDuration="2m12.07639204s" totalImagesPullingTime="2m6.080244087s" totalInitContainerRuntime="3.595487274s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:30:10.073915097 +0000 UTC m=+1809.364529268" watchObservedRunningTime="2026-08-04 18:30:10.07639204 +0000 UTC m=+1809.367006210"
Aug 04 18:30:14.053739 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:14.053701    2566 generic.go:356] "Generic (PLEG): container finished" podID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerID="9b9310caab6a4caefe5354cead99129721ad7efb514b7d51515ed4c8f0de926e" exitCode=0
Aug 04 18:30:14.054287 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:14.053768    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" event={"ID":"00ea82f3-57c6-4de8-a960-37f114fb4170","Type":"ContainerDied","Data":"9b9310caab6a4caefe5354cead99129721ad7efb514b7d51515ed4c8f0de926e"}
Aug 04 18:30:14.055449 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:14.055429    2566 generic.go:356] "Generic (PLEG): container finished" podID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerID="d364e4a108af28fee081751ee9ae0757e3c4b1424e5eff93be3816f655fa3cd4" exitCode=0
Aug 04 18:30:14.055541 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:14.055504    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" event={"ID":"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f","Type":"ContainerDied","Data":"d364e4a108af28fee081751ee9ae0757e3c4b1424e5eff93be3816f655fa3cd4"}
Aug 04 18:30:15.063501 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:15.063466    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" event={"ID":"00ea82f3-57c6-4de8-a960-37f114fb4170","Type":"ContainerStarted","Data":"129936c9dfd4eb8cf2e2f6d86ed007f1e3ab7502f3d426c0bebdf27852e3bc4c"}
Aug 04 18:30:15.063945 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:15.063539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" event={"ID":"00ea82f3-57c6-4de8-a960-37f114fb4170","Type":"ContainerStarted","Data":"140db610680c8ef08d51179edb14b9ee7d31b964abd39406f91edbd7b1e12705"}
Aug 04 18:30:15.063945 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:15.063775    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:15.065674 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:15.065643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" event={"ID":"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f","Type":"ContainerStarted","Data":"7ab6500b2aebd0d64a26a4eebcd57347de0a9776150694de9e5fd9acd33b626e"}
Aug 04 18:30:15.065786 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:15.065696    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" event={"ID":"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f","Type":"ContainerStarted","Data":"d0f7f19eea841f0950708e39e3c30c5ad9d49f061b70a17b5886f598339662d6"}
Aug 04 18:30:15.065961 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:15.065949    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:15.066003 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:15.065985    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:15.067258 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:15.067231    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Aug 04 18:30:15.091183 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:15.091140    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" podStartSLOduration=10.702243441 podStartE2EDuration="15.091091507s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.388848066s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:30:15.089815558 +0000 UTC m=+1814.380429730" watchObservedRunningTime="2026-08-04 18:30:15.091091507 +0000 UTC m=+1814.381705681"
Aug 04 18:30:15.119613 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:15.119563    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" podStartSLOduration=10.991002604 podStartE2EDuration="15.119547895s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.128545291s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:30:15.116126313 +0000 UTC m=+1814.406740481" watchObservedRunningTime="2026-08-04 18:30:15.119547895 +0000 UTC m=+1814.410162065"
Aug 04 18:30:16.041299 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:16.041273    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:30:16.070881 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:16.070840    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Aug 04 18:30:16.071373 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:16.071061    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:16.072272 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:16.072251    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Aug 04 18:30:17.075870 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:17.075837    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Aug 04 18:30:21.075418 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:21.075383    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:30:21.075997 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:21.075935    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Aug 04 18:30:22.080435 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:22.080410    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:30:22.080907 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:22.080878    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Aug 04 18:30:31.075950 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:31.075909    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Aug 04 18:30:32.081083 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:32.081042    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Aug 04 18:30:41.076870 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:41.076825    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Aug 04 18:30:42.081071 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:42.081031    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Aug 04 18:30:46.047013 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:46.046964    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8" podUID="a074d99c-684f-4964-9d30-3b688babff3d" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Aug 04 18:30:51.076023 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:51.075979    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Aug 04 18:30:52.081173 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:52.081133    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Aug 04 18:30:56.044298 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:56.044229    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:30:58.544584 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:58.544546    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"]
Aug 04 18:30:58.545062 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:58.544780    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8" podUID="a074d99c-684f-4964-9d30-3b688babff3d" containerName="kserve-container" containerID="cri-o://f1b0d9a7cf3b04255b8297fc40d708439da6eaadf504efc513b3e444c500bbfd" gracePeriod=30
Aug 04 18:30:58.545062 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:58.544829    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8" podUID="a074d99c-684f-4964-9d30-3b688babff3d" containerName="kube-rbac-proxy" containerID="cri-o://3c47a4ccfece022f300aa26a2d6cc77dd59619cf5d4c9abf9e2c709802e7fd5f" gracePeriod=30
Aug 04 18:30:59.297079 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:59.297046    2566 generic.go:356] "Generic (PLEG): container finished" podID="a074d99c-684f-4964-9d30-3b688babff3d" containerID="3c47a4ccfece022f300aa26a2d6cc77dd59619cf5d4c9abf9e2c709802e7fd5f" exitCode=2
Aug 04 18:30:59.297243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:30:59.297102    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8" event={"ID":"a074d99c-684f-4964-9d30-3b688babff3d","Type":"ContainerDied","Data":"3c47a4ccfece022f300aa26a2d6cc77dd59619cf5d4c9abf9e2c709802e7fd5f"}
Aug 04 18:31:01.036395 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:01.036327    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8" podUID="a074d99c-684f-4964-9d30-3b688babff3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Aug 04 18:31:01.076753 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:01.076716    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Aug 04 18:31:02.081422 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:02.081384    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Aug 04 18:31:03.668518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.668483    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"]
Aug 04 18:31:03.671468 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.671448    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:03.674048 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.674027    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Aug 04 18:31:03.674300 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.674286    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Aug 04 18:31:03.683327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.683304    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"]
Aug 04 18:31:03.802023 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.801991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7bd7699795-ff6wq\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:03.802023 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.802028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxg7f\" (UniqueName: \"kubernetes.io/projected/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kube-api-access-xxg7f\") pod \"isvc-sklearn-runtime-predictor-7bd7699795-ff6wq\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:03.802250 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.802048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7bd7699795-ff6wq\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:03.802250 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.802078    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7bd7699795-ff6wq\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:03.802250 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.802147    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7bd7699795-ff6wq\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:03.902925 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.902879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7bd7699795-ff6wq\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:03.902925 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.902928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxg7f\" (UniqueName: \"kubernetes.io/projected/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kube-api-access-xxg7f\") pod \"isvc-sklearn-runtime-predictor-7bd7699795-ff6wq\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:03.902925 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.902945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7bd7699795-ff6wq\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:03.903263 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:03.903046    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Aug 04 18:31:03.903263 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.903091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7bd7699795-ff6wq\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:03.903263 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:03.903116    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-proxy-tls podName:ca897e64-a9d2-46d5-ab27-cfc6e161dc28 nodeName:}" failed. No retries permitted until 2026-08-04 18:31:04.403099362 +0000 UTC m=+1863.693713511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-proxy-tls") pod "isvc-sklearn-runtime-predictor-7bd7699795-ff6wq" (UID: "ca897e64-a9d2-46d5-ab27-cfc6e161dc28") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Aug 04 18:31:03.903263 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.903188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7bd7699795-ff6wq\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:03.903566 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.903546    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7bd7699795-ff6wq\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:03.903634 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.903598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7bd7699795-ff6wq\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:03.903782 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.903761    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7bd7699795-ff6wq\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:03.913437 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:03.913413    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxg7f\" (UniqueName: \"kubernetes.io/projected/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kube-api-access-xxg7f\") pod \"isvc-sklearn-runtime-predictor-7bd7699795-ff6wq\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:04.408691 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:04.408655    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7bd7699795-ff6wq\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:04.411082 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:04.411054    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7bd7699795-ff6wq\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:04.582550 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:04.582513    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:04.802568 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:04.802538    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"]
Aug 04 18:31:04.804261 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:31:04.804235    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca897e64_a9d2_46d5_ab27_cfc6e161dc28.slice/crio-1b9b6548a5369db2509b7a5fa7cd679c20da3e39afce78368f630092eff23e7f WatchSource:0}: Error finding container 1b9b6548a5369db2509b7a5fa7cd679c20da3e39afce78368f630092eff23e7f: Status 404 returned error can't find the container with id 1b9b6548a5369db2509b7a5fa7cd679c20da3e39afce78368f630092eff23e7f
Aug 04 18:31:05.329296 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:05.329262    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq" event={"ID":"ca897e64-a9d2-46d5-ab27-cfc6e161dc28","Type":"ContainerStarted","Data":"07a54ef0f0dc3062159bd2c3fde5a78ec6093d0a359e333caab84c554c817fa9"}
Aug 04 18:31:05.329485 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:05.329317    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq" event={"ID":"ca897e64-a9d2-46d5-ab27-cfc6e161dc28","Type":"ContainerStarted","Data":"1b9b6548a5369db2509b7a5fa7cd679c20da3e39afce78368f630092eff23e7f"}
Aug 04 18:31:06.035720 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.035680    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8" podUID="a074d99c-684f-4964-9d30-3b688babff3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Aug 04 18:31:06.042167 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.042139    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8" podUID="a074d99c-684f-4964-9d30-3b688babff3d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.134.0.47:8080: connect: connection refused"
Aug 04 18:31:06.528542 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.528517    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:31:06.628588 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.628508    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a074d99c-684f-4964-9d30-3b688babff3d-kserve-provision-location\") pod \"a074d99c-684f-4964-9d30-3b688babff3d\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") "
Aug 04 18:31:06.628588 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.628554    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a074d99c-684f-4964-9d30-3b688babff3d-cabundle-cert\") pod \"a074d99c-684f-4964-9d30-3b688babff3d\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") "
Aug 04 18:31:06.628588 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.628577    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m69t7\" (UniqueName: \"kubernetes.io/projected/a074d99c-684f-4964-9d30-3b688babff3d-kube-api-access-m69t7\") pod \"a074d99c-684f-4964-9d30-3b688babff3d\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") "
Aug 04 18:31:06.628862 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.628607    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a074d99c-684f-4964-9d30-3b688babff3d-kube-rbac-proxy-sar-config\") pod \"a074d99c-684f-4964-9d30-3b688babff3d\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") "
Aug 04 18:31:06.628862 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.628659    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a074d99c-684f-4964-9d30-3b688babff3d-proxy-tls\") pod \"a074d99c-684f-4964-9d30-3b688babff3d\" (UID: \"a074d99c-684f-4964-9d30-3b688babff3d\") "
Aug 04 18:31:06.628976 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.628824    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a074d99c-684f-4964-9d30-3b688babff3d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a074d99c-684f-4964-9d30-3b688babff3d" (UID: "a074d99c-684f-4964-9d30-3b688babff3d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:31:06.628976 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.628922    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a074d99c-684f-4964-9d30-3b688babff3d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a074d99c-684f-4964-9d30-3b688babff3d" (UID: "a074d99c-684f-4964-9d30-3b688babff3d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:31:06.629070 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.629006    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a074d99c-684f-4964-9d30-3b688babff3d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a074d99c-684f-4964-9d30-3b688babff3d" (UID: "a074d99c-684f-4964-9d30-3b688babff3d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:31:06.630688 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.630663    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a074d99c-684f-4964-9d30-3b688babff3d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a074d99c-684f-4964-9d30-3b688babff3d" (UID: "a074d99c-684f-4964-9d30-3b688babff3d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:31:06.630786 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.630727    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a074d99c-684f-4964-9d30-3b688babff3d-kube-api-access-m69t7" (OuterVolumeSpecName: "kube-api-access-m69t7") pod "a074d99c-684f-4964-9d30-3b688babff3d" (UID: "a074d99c-684f-4964-9d30-3b688babff3d"). InnerVolumeSpecName "kube-api-access-m69t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:31:06.729742 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.729705    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a074d99c-684f-4964-9d30-3b688babff3d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:06.729742 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.729737    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a074d99c-684f-4964-9d30-3b688babff3d-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:06.729742 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.729746    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a074d99c-684f-4964-9d30-3b688babff3d-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:06.729981 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.729755    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a074d99c-684f-4964-9d30-3b688babff3d-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:06.729981 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:06.729764    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m69t7\" (UniqueName: \"kubernetes.io/projected/a074d99c-684f-4964-9d30-3b688babff3d-kube-api-access-m69t7\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:07.340054 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:07.340022    2566 generic.go:356] "Generic (PLEG): container finished" podID="a074d99c-684f-4964-9d30-3b688babff3d" containerID="f1b0d9a7cf3b04255b8297fc40d708439da6eaadf504efc513b3e444c500bbfd" exitCode=0
Aug 04 18:31:07.340508 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:07.340080    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"
Aug 04 18:31:07.340508 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:07.340109    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8" event={"ID":"a074d99c-684f-4964-9d30-3b688babff3d","Type":"ContainerDied","Data":"f1b0d9a7cf3b04255b8297fc40d708439da6eaadf504efc513b3e444c500bbfd"}
Aug 04 18:31:07.340508 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:07.340166    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8" event={"ID":"a074d99c-684f-4964-9d30-3b688babff3d","Type":"ContainerDied","Data":"19dc808d4c61f2a9126a0eee249511b5f7703727b20db0abaf2c6ca4f136bc85"}
Aug 04 18:31:07.340508 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:07.340192    2566 scope.go:117] "RemoveContainer" containerID="3c47a4ccfece022f300aa26a2d6cc77dd59619cf5d4c9abf9e2c709802e7fd5f"
Aug 04 18:31:07.350659 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:07.350641    2566 scope.go:117] "RemoveContainer" containerID="f1b0d9a7cf3b04255b8297fc40d708439da6eaadf504efc513b3e444c500bbfd"
Aug 04 18:31:07.358599 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:07.358578    2566 scope.go:117] "RemoveContainer" containerID="46ec0cdccac6b1bd67395c7c91271328e7952a0468d92a52afc5f0dc1b726406"
Aug 04 18:31:07.365442 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:07.365417    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"]
Aug 04 18:31:07.366681 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:07.366669    2566 scope.go:117] "RemoveContainer" containerID="3c47a4ccfece022f300aa26a2d6cc77dd59619cf5d4c9abf9e2c709802e7fd5f"
Aug 04 18:31:07.366965 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:07.366946    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c47a4ccfece022f300aa26a2d6cc77dd59619cf5d4c9abf9e2c709802e7fd5f\": container with ID starting with 3c47a4ccfece022f300aa26a2d6cc77dd59619cf5d4c9abf9e2c709802e7fd5f not found: ID does not exist" containerID="3c47a4ccfece022f300aa26a2d6cc77dd59619cf5d4c9abf9e2c709802e7fd5f"
Aug 04 18:31:07.367046 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:07.366978    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c47a4ccfece022f300aa26a2d6cc77dd59619cf5d4c9abf9e2c709802e7fd5f"} err="failed to get container status \"3c47a4ccfece022f300aa26a2d6cc77dd59619cf5d4c9abf9e2c709802e7fd5f\": rpc error: code = NotFound desc = could not find container \"3c47a4ccfece022f300aa26a2d6cc77dd59619cf5d4c9abf9e2c709802e7fd5f\": container with ID starting with 3c47a4ccfece022f300aa26a2d6cc77dd59619cf5d4c9abf9e2c709802e7fd5f not found: ID does not exist"
Aug 04 18:31:07.367046 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:07.367006    2566 scope.go:117] "RemoveContainer" containerID="f1b0d9a7cf3b04255b8297fc40d708439da6eaadf504efc513b3e444c500bbfd"
Aug 04 18:31:07.367286 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:07.367265    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b0d9a7cf3b04255b8297fc40d708439da6eaadf504efc513b3e444c500bbfd\": container with ID starting with f1b0d9a7cf3b04255b8297fc40d708439da6eaadf504efc513b3e444c500bbfd not found: ID does not exist" containerID="f1b0d9a7cf3b04255b8297fc40d708439da6eaadf504efc513b3e444c500bbfd"
Aug 04 18:31:07.367326 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:07.367292    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b0d9a7cf3b04255b8297fc40d708439da6eaadf504efc513b3e444c500bbfd"} err="failed to get container status \"f1b0d9a7cf3b04255b8297fc40d708439da6eaadf504efc513b3e444c500bbfd\": rpc error: code = NotFound desc = could not find container \"f1b0d9a7cf3b04255b8297fc40d708439da6eaadf504efc513b3e444c500bbfd\": container with ID starting with f1b0d9a7cf3b04255b8297fc40d708439da6eaadf504efc513b3e444c500bbfd not found: ID does not exist"
Aug 04 18:31:07.367326 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:07.367309    2566 scope.go:117] "RemoveContainer" containerID="46ec0cdccac6b1bd67395c7c91271328e7952a0468d92a52afc5f0dc1b726406"
Aug 04 18:31:07.367571 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:07.367551    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ec0cdccac6b1bd67395c7c91271328e7952a0468d92a52afc5f0dc1b726406\": container with ID starting with 46ec0cdccac6b1bd67395c7c91271328e7952a0468d92a52afc5f0dc1b726406 not found: ID does not exist" containerID="46ec0cdccac6b1bd67395c7c91271328e7952a0468d92a52afc5f0dc1b726406"
Aug 04 18:31:07.367630 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:07.367578    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ec0cdccac6b1bd67395c7c91271328e7952a0468d92a52afc5f0dc1b726406"} err="failed to get container status \"46ec0cdccac6b1bd67395c7c91271328e7952a0468d92a52afc5f0dc1b726406\": rpc error: code = NotFound desc = could not find container \"46ec0cdccac6b1bd67395c7c91271328e7952a0468d92a52afc5f0dc1b726406\": container with ID starting with 46ec0cdccac6b1bd67395c7c91271328e7952a0468d92a52afc5f0dc1b726406 not found: ID does not exist"
Aug 04 18:31:07.373889 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:07.373867    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-jpsc8"]
Aug 04 18:31:09.196950 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:09.196919    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a074d99c-684f-4964-9d30-3b688babff3d" path="/var/lib/kubelet/pods/a074d99c-684f-4964-9d30-3b688babff3d/volumes"
Aug 04 18:31:11.076148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:11.076106    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Aug 04 18:31:11.365951 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:11.365873    2566 generic.go:356] "Generic (PLEG): container finished" podID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" containerID="07a54ef0f0dc3062159bd2c3fde5a78ec6093d0a359e333caab84c554c817fa9" exitCode=0
Aug 04 18:31:11.365951 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:11.365932    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq" event={"ID":"ca897e64-a9d2-46d5-ab27-cfc6e161dc28","Type":"ContainerDied","Data":"07a54ef0f0dc3062159bd2c3fde5a78ec6093d0a359e333caab84c554c817fa9"}
Aug 04 18:31:12.081268 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:12.081228    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Aug 04 18:31:12.375474 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:12.375383    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq" event={"ID":"ca897e64-a9d2-46d5-ab27-cfc6e161dc28","Type":"ContainerStarted","Data":"a36a5b6e074c80209459af85b1a194ac826fa1f1fbfe18c15a4114cf0beaebfa"}
Aug 04 18:31:12.375474 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:12.375451    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq" event={"ID":"ca897e64-a9d2-46d5-ab27-cfc6e161dc28","Type":"ContainerStarted","Data":"65dbffaa401055019361ee7994b4a2d6a7f7c8a4e578914c08932750b14fd174"}
Aug 04 18:31:12.375723 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:12.375690    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:12.375889 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:12.375750    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:12.377345 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:12.377319    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq" podUID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Aug 04 18:31:12.402153 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:12.402091    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq" podStartSLOduration=3.557077119 podStartE2EDuration="9.402073456s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.844996337s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:31:12.400070195 +0000 UTC m=+1871.690684389" watchObservedRunningTime="2026-08-04 18:31:12.402073456 +0000 UTC m=+1871.692687627"
Aug 04 18:31:13.380150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:13.380094    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq" podUID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Aug 04 18:31:18.385050 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:18.385019    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:18.385593 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:18.385564    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq" podUID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Aug 04 18:31:21.076130 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:21.076089    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Aug 04 18:31:22.080979 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:22.080938    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Aug 04 18:31:28.386159 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:28.386081    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:31.076557 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:31.076508    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Aug 04 18:31:32.081861 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:32.081823    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Aug 04 18:31:36.066950 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:36.066918    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"]
Aug 04 18:31:36.067369 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:36.067169    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq" podUID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" containerName="kserve-container" containerID="cri-o://65dbffaa401055019361ee7994b4a2d6a7f7c8a4e578914c08932750b14fd174" gracePeriod=30
Aug 04 18:31:36.067369 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:36.067228    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq" podUID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" containerName="kube-rbac-proxy" containerID="cri-o://a36a5b6e074c80209459af85b1a194ac826fa1f1fbfe18c15a4114cf0beaebfa" gracePeriod=30
Aug 04 18:31:36.514039 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:36.513955    2566 generic.go:356] "Generic (PLEG): container finished" podID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" containerID="a36a5b6e074c80209459af85b1a194ac826fa1f1fbfe18c15a4114cf0beaebfa" exitCode=2
Aug 04 18:31:36.514039 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:36.513995    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq" event={"ID":"ca897e64-a9d2-46d5-ab27-cfc6e161dc28","Type":"ContainerDied","Data":"a36a5b6e074c80209459af85b1a194ac826fa1f1fbfe18c15a4114cf0beaebfa"}
Aug 04 18:31:37.075192 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.075170    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:37.218992 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.218958    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kserve-provision-location\") pod \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") "
Aug 04 18:31:37.218992 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.219000    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kube-rbac-proxy-sar-config\") pod \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") "
Aug 04 18:31:37.219203 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.219021    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-cabundle-cert\") pod \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") "
Aug 04 18:31:37.219203 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.219142    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxg7f\" (UniqueName: \"kubernetes.io/projected/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kube-api-access-xxg7f\") pod \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") "
Aug 04 18:31:37.219203 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.219185    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-proxy-tls\") pod \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\" (UID: \"ca897e64-a9d2-46d5-ab27-cfc6e161dc28\") "
Aug 04 18:31:37.219442 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.219417    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca897e64-a9d2-46d5-ab27-cfc6e161dc28" (UID: "ca897e64-a9d2-46d5-ab27-cfc6e161dc28"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:31:37.219442 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.219425    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca897e64-a9d2-46d5-ab27-cfc6e161dc28" (UID: "ca897e64-a9d2-46d5-ab27-cfc6e161dc28"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:31:37.219559 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.219498    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:37.219559 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.219512    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:37.221299 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.221266    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca897e64-a9d2-46d5-ab27-cfc6e161dc28" (UID: "ca897e64-a9d2-46d5-ab27-cfc6e161dc28"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:31:37.221299 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.221279    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kube-api-access-xxg7f" (OuterVolumeSpecName: "kube-api-access-xxg7f") pod "ca897e64-a9d2-46d5-ab27-cfc6e161dc28" (UID: "ca897e64-a9d2-46d5-ab27-cfc6e161dc28"). InnerVolumeSpecName "kube-api-access-xxg7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:31:37.246313 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.246272    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca897e64-a9d2-46d5-ab27-cfc6e161dc28" (UID: "ca897e64-a9d2-46d5-ab27-cfc6e161dc28"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:31:37.320137 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.320058    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:37.320137 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.320085    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxg7f\" (UniqueName: \"kubernetes.io/projected/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-kube-api-access-xxg7f\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:37.320137 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.320095    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca897e64-a9d2-46d5-ab27-cfc6e161dc28-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:37.520422 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.520387    2566 generic.go:356] "Generic (PLEG): container finished" podID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" containerID="65dbffaa401055019361ee7994b4a2d6a7f7c8a4e578914c08932750b14fd174" exitCode=0
Aug 04 18:31:37.520624 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.520438    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq" event={"ID":"ca897e64-a9d2-46d5-ab27-cfc6e161dc28","Type":"ContainerDied","Data":"65dbffaa401055019361ee7994b4a2d6a7f7c8a4e578914c08932750b14fd174"}
Aug 04 18:31:37.520624 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.520451    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"
Aug 04 18:31:37.520624 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.520490    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq" event={"ID":"ca897e64-a9d2-46d5-ab27-cfc6e161dc28","Type":"ContainerDied","Data":"1b9b6548a5369db2509b7a5fa7cd679c20da3e39afce78368f630092eff23e7f"}
Aug 04 18:31:37.520624 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.520519    2566 scope.go:117] "RemoveContainer" containerID="a36a5b6e074c80209459af85b1a194ac826fa1f1fbfe18c15a4114cf0beaebfa"
Aug 04 18:31:37.529644 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.529621    2566 scope.go:117] "RemoveContainer" containerID="65dbffaa401055019361ee7994b4a2d6a7f7c8a4e578914c08932750b14fd174"
Aug 04 18:31:37.537569 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.537553    2566 scope.go:117] "RemoveContainer" containerID="07a54ef0f0dc3062159bd2c3fde5a78ec6093d0a359e333caab84c554c817fa9"
Aug 04 18:31:37.546542 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.546340    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"]
Aug 04 18:31:37.547432 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.547414    2566 scope.go:117] "RemoveContainer" containerID="a36a5b6e074c80209459af85b1a194ac826fa1f1fbfe18c15a4114cf0beaebfa"
Aug 04 18:31:37.547692 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:37.547673    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a36a5b6e074c80209459af85b1a194ac826fa1f1fbfe18c15a4114cf0beaebfa\": container with ID starting with a36a5b6e074c80209459af85b1a194ac826fa1f1fbfe18c15a4114cf0beaebfa not found: ID does not exist" containerID="a36a5b6e074c80209459af85b1a194ac826fa1f1fbfe18c15a4114cf0beaebfa"
Aug 04 18:31:37.547777 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.547704    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a36a5b6e074c80209459af85b1a194ac826fa1f1fbfe18c15a4114cf0beaebfa"} err="failed to get container status \"a36a5b6e074c80209459af85b1a194ac826fa1f1fbfe18c15a4114cf0beaebfa\": rpc error: code = NotFound desc = could not find container \"a36a5b6e074c80209459af85b1a194ac826fa1f1fbfe18c15a4114cf0beaebfa\": container with ID starting with a36a5b6e074c80209459af85b1a194ac826fa1f1fbfe18c15a4114cf0beaebfa not found: ID does not exist"
Aug 04 18:31:37.547777 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.547728    2566 scope.go:117] "RemoveContainer" containerID="65dbffaa401055019361ee7994b4a2d6a7f7c8a4e578914c08932750b14fd174"
Aug 04 18:31:37.547992 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:37.547962    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65dbffaa401055019361ee7994b4a2d6a7f7c8a4e578914c08932750b14fd174\": container with ID starting with 65dbffaa401055019361ee7994b4a2d6a7f7c8a4e578914c08932750b14fd174 not found: ID does not exist" containerID="65dbffaa401055019361ee7994b4a2d6a7f7c8a4e578914c08932750b14fd174"
Aug 04 18:31:37.548148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.547989    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65dbffaa401055019361ee7994b4a2d6a7f7c8a4e578914c08932750b14fd174"} err="failed to get container status \"65dbffaa401055019361ee7994b4a2d6a7f7c8a4e578914c08932750b14fd174\": rpc error: code = NotFound desc = could not find container \"65dbffaa401055019361ee7994b4a2d6a7f7c8a4e578914c08932750b14fd174\": container with ID starting with 65dbffaa401055019361ee7994b4a2d6a7f7c8a4e578914c08932750b14fd174 not found: ID does not exist"
Aug 04 18:31:37.548148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.548010    2566 scope.go:117] "RemoveContainer" containerID="07a54ef0f0dc3062159bd2c3fde5a78ec6093d0a359e333caab84c554c817fa9"
Aug 04 18:31:37.548330 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:37.548266    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a54ef0f0dc3062159bd2c3fde5a78ec6093d0a359e333caab84c554c817fa9\": container with ID starting with 07a54ef0f0dc3062159bd2c3fde5a78ec6093d0a359e333caab84c554c817fa9 not found: ID does not exist" containerID="07a54ef0f0dc3062159bd2c3fde5a78ec6093d0a359e333caab84c554c817fa9"
Aug 04 18:31:37.548330 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.548294    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a54ef0f0dc3062159bd2c3fde5a78ec6093d0a359e333caab84c554c817fa9"} err="failed to get container status \"07a54ef0f0dc3062159bd2c3fde5a78ec6093d0a359e333caab84c554c817fa9\": rpc error: code = NotFound desc = could not find container \"07a54ef0f0dc3062159bd2c3fde5a78ec6093d0a359e333caab84c554c817fa9\": container with ID starting with 07a54ef0f0dc3062159bd2c3fde5a78ec6093d0a359e333caab84c554c817fa9 not found: ID does not exist"
Aug 04 18:31:37.550717 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:37.550698    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7bd7699795-ff6wq"]
Aug 04 18:31:39.194262 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:39.194217    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" path="/var/lib/kubelet/pods/ca897e64-a9d2-46d5-ab27-cfc6e161dc28/volumes"
Aug 04 18:31:41.076505 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.076472    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:31:41.223542 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.223512    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"]
Aug 04 18:31:41.223887 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.223875    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a074d99c-684f-4964-9d30-3b688babff3d" containerName="kube-rbac-proxy"
Aug 04 18:31:41.223937 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.223888    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a074d99c-684f-4964-9d30-3b688babff3d" containerName="kube-rbac-proxy"
Aug 04 18:31:41.223937 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.223897    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" containerName="kube-rbac-proxy"
Aug 04 18:31:41.223937 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.223904    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" containerName="kube-rbac-proxy"
Aug 04 18:31:41.223937 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.223928    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" containerName="storage-initializer"
Aug 04 18:31:41.223937 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.223934    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" containerName="storage-initializer"
Aug 04 18:31:41.224085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.223944    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" containerName="kserve-container"
Aug 04 18:31:41.224085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.223950    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" containerName="kserve-container"
Aug 04 18:31:41.224085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.223959    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a074d99c-684f-4964-9d30-3b688babff3d" containerName="storage-initializer"
Aug 04 18:31:41.224085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.223964    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a074d99c-684f-4964-9d30-3b688babff3d" containerName="storage-initializer"
Aug 04 18:31:41.224085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.223971    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a074d99c-684f-4964-9d30-3b688babff3d" containerName="kserve-container"
Aug 04 18:31:41.224085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.223976    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a074d99c-684f-4964-9d30-3b688babff3d" containerName="kserve-container"
Aug 04 18:31:41.224085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.224029    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" containerName="kube-rbac-proxy"
Aug 04 18:31:41.224085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.224038    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a074d99c-684f-4964-9d30-3b688babff3d" containerName="kube-rbac-proxy"
Aug 04 18:31:41.224085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.224045    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca897e64-a9d2-46d5-ab27-cfc6e161dc28" containerName="kserve-container"
Aug 04 18:31:41.224085 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.224051    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a074d99c-684f-4964-9d30-3b688babff3d" containerName="kserve-container"
Aug 04 18:31:41.227584 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.227566    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:41.229899 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.229873    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Aug 04 18:31:41.230075 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.230062    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Aug 04 18:31:41.237774 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.237753    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"]
Aug 04 18:31:41.354625 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.354531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx2km\" (UniqueName: \"kubernetes.io/projected/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kube-api-access-sx2km\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:41.354625 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.354574    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5754703d-ce56-4877-ad5c-b26eb4ce32ee-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:41.354815 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.354658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:41.354815 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.354722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5754703d-ce56-4877-ad5c-b26eb4ce32ee-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:41.354815 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.354764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:41.455750 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.455715    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx2km\" (UniqueName: \"kubernetes.io/projected/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kube-api-access-sx2km\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:41.455950 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.455770    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5754703d-ce56-4877-ad5c-b26eb4ce32ee-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:41.455950 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.455837    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:41.455950 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.455917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5754703d-ce56-4877-ad5c-b26eb4ce32ee-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:41.456134 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.455961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:41.456134 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:41.456041    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Aug 04 18:31:41.456240 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:41.456177    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5754703d-ce56-4877-ad5c-b26eb4ce32ee-proxy-tls podName:5754703d-ce56-4877-ad5c-b26eb4ce32ee nodeName:}" failed. No retries permitted until 2026-08-04 18:31:41.956115333 +0000 UTC m=+1901.246729495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5754703d-ce56-4877-ad5c-b26eb4ce32ee-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6" (UID: "5754703d-ce56-4877-ad5c-b26eb4ce32ee") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Aug 04 18:31:41.456398 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.456375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:41.456601 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.456580    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:41.456677 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.456648    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5754703d-ce56-4877-ad5c-b26eb4ce32ee-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:41.465330 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.465300    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx2km\" (UniqueName: \"kubernetes.io/projected/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kube-api-access-sx2km\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:41.960617 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.960586    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5754703d-ce56-4877-ad5c-b26eb4ce32ee-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:41.963146 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:41.963117    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5754703d-ce56-4877-ad5c-b26eb4ce32ee-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:42.081409 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:42.081349    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:31:42.138545 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:42.138510    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:42.275119 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:42.275097    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"]
Aug 04 18:31:42.277222 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:31:42.277195    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5754703d_ce56_4877_ad5c_b26eb4ce32ee.slice/crio-78c6b80ef11b623c81da292f1cc6b8ef617a580a3b79adda25c0e06c74c1df54 WatchSource:0}: Error finding container 78c6b80ef11b623c81da292f1cc6b8ef617a580a3b79adda25c0e06c74c1df54: Status 404 returned error can't find the container with id 78c6b80ef11b623c81da292f1cc6b8ef617a580a3b79adda25c0e06c74c1df54
Aug 04 18:31:42.547148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:42.547068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6" event={"ID":"5754703d-ce56-4877-ad5c-b26eb4ce32ee","Type":"ContainerStarted","Data":"358d4807d8935d32fa54b7d965dcdc504c411b6d4e8f19ce3a6b09a52d5fce73"}
Aug 04 18:31:42.547148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:42.547123    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6" event={"ID":"5754703d-ce56-4877-ad5c-b26eb4ce32ee","Type":"ContainerStarted","Data":"78c6b80ef11b623c81da292f1cc6b8ef617a580a3b79adda25c0e06c74c1df54"}
Aug 04 18:31:46.569014 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:46.568980    2566 generic.go:356] "Generic (PLEG): container finished" podID="5754703d-ce56-4877-ad5c-b26eb4ce32ee" containerID="358d4807d8935d32fa54b7d965dcdc504c411b6d4e8f19ce3a6b09a52d5fce73" exitCode=0
Aug 04 18:31:46.569408 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:46.569052    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6" event={"ID":"5754703d-ce56-4877-ad5c-b26eb4ce32ee","Type":"ContainerDied","Data":"358d4807d8935d32fa54b7d965dcdc504c411b6d4e8f19ce3a6b09a52d5fce73"}
Aug 04 18:31:47.575682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:47.575646    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6" event={"ID":"5754703d-ce56-4877-ad5c-b26eb4ce32ee","Type":"ContainerStarted","Data":"e24f41a16e6c27d8831cf911c45ede25292edd1e5f87af2bfb3db3dd660eec0b"}
Aug 04 18:31:47.576062 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:47.575711    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6" event={"ID":"5754703d-ce56-4877-ad5c-b26eb4ce32ee","Type":"ContainerStarted","Data":"bae3aa4c926ef0863f76556c00958010eedf5d3a11a3f811cdc9bfadb6e80e9a"}
Aug 04 18:31:47.576062 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:47.575910    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:47.602804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:47.602742    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6" podStartSLOduration=2.832478563 podStartE2EDuration="6.60272153s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.770242967s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:31:47.600911022 +0000 UTC m=+1906.891525207" watchObservedRunningTime="2026-08-04 18:31:47.60272153 +0000 UTC m=+1906.893335700"
Aug 04 18:31:48.580431 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:48.580396    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:50.162232 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:50.162198    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"]
Aug 04 18:31:50.162700 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:50.162510    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kserve-container" containerID="cri-o://140db610680c8ef08d51179edb14b9ee7d31b964abd39406f91edbd7b1e12705" gracePeriod=30
Aug 04 18:31:50.162700 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:50.162572    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kube-rbac-proxy" containerID="cri-o://129936c9dfd4eb8cf2e2f6d86ed007f1e3ab7502f3d426c0bebdf27852e3bc4c" gracePeriod=30
Aug 04 18:31:50.164595 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:50.164572    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"]
Aug 04 18:31:50.164808 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:50.164782    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kserve-container" containerID="cri-o://d0f7f19eea841f0950708e39e3c30c5ad9d49f061b70a17b5886f598339662d6" gracePeriod=30
Aug 04 18:31:50.164984 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:50.164811    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kube-rbac-proxy" containerID="cri-o://7ab6500b2aebd0d64a26a4eebcd57347de0a9776150694de9e5fd9acd33b626e" gracePeriod=30
Aug 04 18:31:50.591441 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:50.591402    2566 generic.go:356] "Generic (PLEG): container finished" podID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerID="129936c9dfd4eb8cf2e2f6d86ed007f1e3ab7502f3d426c0bebdf27852e3bc4c" exitCode=2
Aug 04 18:31:50.591629 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:50.591466    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" event={"ID":"00ea82f3-57c6-4de8-a960-37f114fb4170","Type":"ContainerDied","Data":"129936c9dfd4eb8cf2e2f6d86ed007f1e3ab7502f3d426c0bebdf27852e3bc4c"}
Aug 04 18:31:50.593368 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:50.593336    2566 generic.go:356] "Generic (PLEG): container finished" podID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerID="7ab6500b2aebd0d64a26a4eebcd57347de0a9776150694de9e5fd9acd33b626e" exitCode=2
Aug 04 18:31:50.593470 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:50.593377    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" event={"ID":"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f","Type":"ContainerDied","Data":"7ab6500b2aebd0d64a26a4eebcd57347de0a9776150694de9e5fd9acd33b626e"}
Aug 04 18:31:51.072142 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:51.072099    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Aug 04 18:31:51.076552 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:51.076527    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Aug 04 18:31:52.076652 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:52.076614    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Aug 04 18:31:52.080912 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:52.080887    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Aug 04 18:31:54.591856 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:54.591832    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:31:55.116888 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.116858    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:31:55.185006 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.184975    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00ea82f3-57c6-4de8-a960-37f114fb4170-kube-rbac-proxy-sar-config\") pod \"00ea82f3-57c6-4de8-a960-37f114fb4170\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") "
Aug 04 18:31:55.185168 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.185014    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00ea82f3-57c6-4de8-a960-37f114fb4170-cabundle-cert\") pod \"00ea82f3-57c6-4de8-a960-37f114fb4170\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") "
Aug 04 18:31:55.185168 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.185048    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwbw7\" (UniqueName: \"kubernetes.io/projected/00ea82f3-57c6-4de8-a960-37f114fb4170-kube-api-access-gwbw7\") pod \"00ea82f3-57c6-4de8-a960-37f114fb4170\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") "
Aug 04 18:31:55.185168 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.185088    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00ea82f3-57c6-4de8-a960-37f114fb4170-kserve-provision-location\") pod \"00ea82f3-57c6-4de8-a960-37f114fb4170\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") "
Aug 04 18:31:55.185168 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.185119    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00ea82f3-57c6-4de8-a960-37f114fb4170-proxy-tls\") pod \"00ea82f3-57c6-4de8-a960-37f114fb4170\" (UID: \"00ea82f3-57c6-4de8-a960-37f114fb4170\") "
Aug 04 18:31:55.185410 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.185383    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ea82f3-57c6-4de8-a960-37f114fb4170-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "00ea82f3-57c6-4de8-a960-37f114fb4170" (UID: "00ea82f3-57c6-4de8-a960-37f114fb4170"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:31:55.185451 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.185417    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ea82f3-57c6-4de8-a960-37f114fb4170-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "00ea82f3-57c6-4de8-a960-37f114fb4170" (UID: "00ea82f3-57c6-4de8-a960-37f114fb4170"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:31:55.185485 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.185470    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ea82f3-57c6-4de8-a960-37f114fb4170-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "00ea82f3-57c6-4de8-a960-37f114fb4170" (UID: "00ea82f3-57c6-4de8-a960-37f114fb4170"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:31:55.187254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.187225    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ea82f3-57c6-4de8-a960-37f114fb4170-kube-api-access-gwbw7" (OuterVolumeSpecName: "kube-api-access-gwbw7") pod "00ea82f3-57c6-4de8-a960-37f114fb4170" (UID: "00ea82f3-57c6-4de8-a960-37f114fb4170"). InnerVolumeSpecName "kube-api-access-gwbw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:31:55.187439 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.187227    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00ea82f3-57c6-4de8-a960-37f114fb4170-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00ea82f3-57c6-4de8-a960-37f114fb4170" (UID: "00ea82f3-57c6-4de8-a960-37f114fb4170"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:31:55.248925 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.248896    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:31:55.286178 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.286152    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-proxy-tls\") pod \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") "
Aug 04 18:31:55.286391 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.286186    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgs7t\" (UniqueName: \"kubernetes.io/projected/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kube-api-access-cgs7t\") pod \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") "
Aug 04 18:31:55.286391 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.286228    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kube-rbac-proxy-sar-config\") pod \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") "
Aug 04 18:31:55.286391 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.286276    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-cabundle-cert\") pod \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") "
Aug 04 18:31:55.286391 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.286326    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kserve-provision-location\") pod \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\" (UID: \"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f\") "
Aug 04 18:31:55.286665 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.286606    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00ea82f3-57c6-4de8-a960-37f114fb4170-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:55.286665 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.286625    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00ea82f3-57c6-4de8-a960-37f114fb4170-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:55.286665 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.286640    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwbw7\" (UniqueName: \"kubernetes.io/projected/00ea82f3-57c6-4de8-a960-37f114fb4170-kube-api-access-gwbw7\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:55.286665 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.286660    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00ea82f3-57c6-4de8-a960-37f114fb4170-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:55.286849 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.286674    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00ea82f3-57c6-4de8-a960-37f114fb4170-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:55.286849 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.286676    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" (UID: "67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:31:55.286849 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.286779    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" (UID: "67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:31:55.287787 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.286920    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" (UID: "67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:31:55.289396 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.289343    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" (UID: "67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:31:55.289554 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.289529    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kube-api-access-cgs7t" (OuterVolumeSpecName: "kube-api-access-cgs7t") pod "67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" (UID: "67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f"). InnerVolumeSpecName "kube-api-access-cgs7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:31:55.372112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372081    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"]
Aug 04 18:31:55.372522 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372505    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kube-rbac-proxy"
Aug 04 18:31:55.372522 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372523    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kube-rbac-proxy"
Aug 04 18:31:55.372770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372538    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kserve-container"
Aug 04 18:31:55.372770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372543    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kserve-container"
Aug 04 18:31:55.372770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372550    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kserve-container"
Aug 04 18:31:55.372770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372555    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kserve-container"
Aug 04 18:31:55.372770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372569    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="storage-initializer"
Aug 04 18:31:55.372770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372577    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="storage-initializer"
Aug 04 18:31:55.372770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372596    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="storage-initializer"
Aug 04 18:31:55.372770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372604    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="storage-initializer"
Aug 04 18:31:55.372770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372624    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kube-rbac-proxy"
Aug 04 18:31:55.372770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372632    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kube-rbac-proxy"
Aug 04 18:31:55.372770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372692    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kserve-container"
Aug 04 18:31:55.372770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372705    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kserve-container"
Aug 04 18:31:55.372770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372714    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerName="kube-rbac-proxy"
Aug 04 18:31:55.372770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.372725    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerName="kube-rbac-proxy"
Aug 04 18:31:55.375331 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.375316    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:55.377662 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.377642    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Aug 04 18:31:55.377780 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.377649    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Aug 04 18:31:55.387002 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.386982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-proxy-tls\") pod \"isvc-canary-promote-predictor-79fd8cbff8-2kc5w\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:55.387112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.387009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-cabundle-cert\") pod \"isvc-canary-promote-predictor-79fd8cbff8-2kc5w\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:55.387112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.387037    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-79fd8cbff8-2kc5w\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:55.387112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.387097    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kserve-provision-location\") pod \"isvc-canary-promote-predictor-79fd8cbff8-2kc5w\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:55.387288 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.387222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bxfz\" (UniqueName: \"kubernetes.io/projected/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kube-api-access-8bxfz\") pod \"isvc-canary-promote-predictor-79fd8cbff8-2kc5w\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:55.387288 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.387281    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:55.387407 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.387298    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:55.387407 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.387313    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:55.387407 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.387327    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgs7t\" (UniqueName: \"kubernetes.io/projected/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kube-api-access-cgs7t\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:55.387407 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.387340    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:31:55.396538 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.396502    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"]
Aug 04 18:31:55.399846 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.399828    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"]
Aug 04 18:31:55.399972 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.399959    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.402827 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.402807    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Aug 04 18:31:55.416005 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.415984    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"]
Aug 04 18:31:55.488095 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.488010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a67838d-68a3-4750-b04d-282d5c375e15-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.488095 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.488052    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bxfz\" (UniqueName: \"kubernetes.io/projected/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kube-api-access-8bxfz\") pod \"isvc-canary-promote-predictor-79fd8cbff8-2kc5w\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:55.488095 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.488085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7zbp\" (UniqueName: \"kubernetes.io/projected/3a67838d-68a3-4750-b04d-282d5c375e15-kube-api-access-z7zbp\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.488314 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.488166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-proxy-tls\") pod \"isvc-canary-promote-predictor-79fd8cbff8-2kc5w\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:55.488314 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.488199    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a67838d-68a3-4750-b04d-282d5c375e15-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.488314 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.488222    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-cabundle-cert\") pod \"isvc-canary-promote-predictor-79fd8cbff8-2kc5w\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:55.488314 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.488267    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a67838d-68a3-4750-b04d-282d5c375e15-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.488314 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.488298    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a67838d-68a3-4750-b04d-282d5c375e15-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.488530 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:55.488314    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Aug 04 18:31:55.488530 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.488343    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-79fd8cbff8-2kc5w\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:55.488530 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:55.488408    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-proxy-tls podName:38e7f1f4-8004-4f57-907a-9b04e2af2a6e nodeName:}" failed. No retries permitted until 2026-08-04 18:31:55.988386253 +0000 UTC m=+1915.279000420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-proxy-tls") pod "isvc-canary-promote-predictor-79fd8cbff8-2kc5w" (UID: "38e7f1f4-8004-4f57-907a-9b04e2af2a6e") : secret "isvc-canary-promote-predictor-serving-cert" not found
Aug 04 18:31:55.488530 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.488446    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kserve-provision-location\") pod \"isvc-canary-promote-predictor-79fd8cbff8-2kc5w\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:55.488792 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.488771    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kserve-provision-location\") pod \"isvc-canary-promote-predictor-79fd8cbff8-2kc5w\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:55.488898 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.488883    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-cabundle-cert\") pod \"isvc-canary-promote-predictor-79fd8cbff8-2kc5w\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:55.488959 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.488943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-79fd8cbff8-2kc5w\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:55.511992 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.511959    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bxfz\" (UniqueName: \"kubernetes.io/projected/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kube-api-access-8bxfz\") pod \"isvc-canary-promote-predictor-79fd8cbff8-2kc5w\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:55.589321 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.589286    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a67838d-68a3-4750-b04d-282d5c375e15-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.589517 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.589330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7zbp\" (UniqueName: \"kubernetes.io/projected/3a67838d-68a3-4750-b04d-282d5c375e15-kube-api-access-z7zbp\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.589517 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.589404    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a67838d-68a3-4750-b04d-282d5c375e15-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.589517 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.589433    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a67838d-68a3-4750-b04d-282d5c375e15-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.589692 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.589530    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a67838d-68a3-4750-b04d-282d5c375e15-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.589772 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.589745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a67838d-68a3-4750-b04d-282d5c375e15-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.590112 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.590093    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a67838d-68a3-4750-b04d-282d5c375e15-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.590202 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.590171    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a67838d-68a3-4750-b04d-282d5c375e15-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.591758 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.591742    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a67838d-68a3-4750-b04d-282d5c375e15-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.605199 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.605178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7zbp\" (UniqueName: \"kubernetes.io/projected/3a67838d-68a3-4750-b04d-282d5c375e15-kube-api-access-z7zbp\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.618549 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.618523    2566 generic.go:356] "Generic (PLEG): container finished" podID="00ea82f3-57c6-4de8-a960-37f114fb4170" containerID="140db610680c8ef08d51179edb14b9ee7d31b964abd39406f91edbd7b1e12705" exitCode=0
Aug 04 18:31:55.618659 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.618582    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"
Aug 04 18:31:55.618659 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.618596    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" event={"ID":"00ea82f3-57c6-4de8-a960-37f114fb4170","Type":"ContainerDied","Data":"140db610680c8ef08d51179edb14b9ee7d31b964abd39406f91edbd7b1e12705"}
Aug 04 18:31:55.618659 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.618646    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7" event={"ID":"00ea82f3-57c6-4de8-a960-37f114fb4170","Type":"ContainerDied","Data":"4f40c34f1720b159ad440915ce691474a4eca0e81ffd8982af61ad09c4b748f8"}
Aug 04 18:31:55.618805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.618675    2566 scope.go:117] "RemoveContainer" containerID="129936c9dfd4eb8cf2e2f6d86ed007f1e3ab7502f3d426c0bebdf27852e3bc4c"
Aug 04 18:31:55.621504 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.621340    2566 generic.go:356] "Generic (PLEG): container finished" podID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" containerID="d0f7f19eea841f0950708e39e3c30c5ad9d49f061b70a17b5886f598339662d6" exitCode=0
Aug 04 18:31:55.621630 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.621539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" event={"ID":"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f","Type":"ContainerDied","Data":"d0f7f19eea841f0950708e39e3c30c5ad9d49f061b70a17b5886f598339662d6"}
Aug 04 18:31:55.621630 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.621588    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"
Aug 04 18:31:55.621822 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.621790    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl" event={"ID":"67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f","Type":"ContainerDied","Data":"5c45a83d7d366674cd8e09dff822d562d441730377916fd793d714fd048e0cd3"}
Aug 04 18:31:55.629389 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.629243    2566 scope.go:117] "RemoveContainer" containerID="140db610680c8ef08d51179edb14b9ee7d31b964abd39406f91edbd7b1e12705"
Aug 04 18:31:55.638595 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.638581    2566 scope.go:117] "RemoveContainer" containerID="9b9310caab6a4caefe5354cead99129721ad7efb514b7d51515ed4c8f0de926e"
Aug 04 18:31:55.646427 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.646403    2566 scope.go:117] "RemoveContainer" containerID="129936c9dfd4eb8cf2e2f6d86ed007f1e3ab7502f3d426c0bebdf27852e3bc4c"
Aug 04 18:31:55.647180 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:55.646723    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"129936c9dfd4eb8cf2e2f6d86ed007f1e3ab7502f3d426c0bebdf27852e3bc4c\": container with ID starting with 129936c9dfd4eb8cf2e2f6d86ed007f1e3ab7502f3d426c0bebdf27852e3bc4c not found: ID does not exist" containerID="129936c9dfd4eb8cf2e2f6d86ed007f1e3ab7502f3d426c0bebdf27852e3bc4c"
Aug 04 18:31:55.647180 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.646892    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"129936c9dfd4eb8cf2e2f6d86ed007f1e3ab7502f3d426c0bebdf27852e3bc4c"} err="failed to get container status \"129936c9dfd4eb8cf2e2f6d86ed007f1e3ab7502f3d426c0bebdf27852e3bc4c\": rpc error: code = NotFound desc = could not find container \"129936c9dfd4eb8cf2e2f6d86ed007f1e3ab7502f3d426c0bebdf27852e3bc4c\": container with ID starting with 129936c9dfd4eb8cf2e2f6d86ed007f1e3ab7502f3d426c0bebdf27852e3bc4c not found: ID does not exist"
Aug 04 18:31:55.647180 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.646920    2566 scope.go:117] "RemoveContainer" containerID="140db610680c8ef08d51179edb14b9ee7d31b964abd39406f91edbd7b1e12705"
Aug 04 18:31:55.647439 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:55.647256    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"140db610680c8ef08d51179edb14b9ee7d31b964abd39406f91edbd7b1e12705\": container with ID starting with 140db610680c8ef08d51179edb14b9ee7d31b964abd39406f91edbd7b1e12705 not found: ID does not exist" containerID="140db610680c8ef08d51179edb14b9ee7d31b964abd39406f91edbd7b1e12705"
Aug 04 18:31:55.647439 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.647288    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"140db610680c8ef08d51179edb14b9ee7d31b964abd39406f91edbd7b1e12705"} err="failed to get container status \"140db610680c8ef08d51179edb14b9ee7d31b964abd39406f91edbd7b1e12705\": rpc error: code = NotFound desc = could not find container \"140db610680c8ef08d51179edb14b9ee7d31b964abd39406f91edbd7b1e12705\": container with ID starting with 140db610680c8ef08d51179edb14b9ee7d31b964abd39406f91edbd7b1e12705 not found: ID does not exist"
Aug 04 18:31:55.647439 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.647340    2566 scope.go:117] "RemoveContainer" containerID="9b9310caab6a4caefe5354cead99129721ad7efb514b7d51515ed4c8f0de926e"
Aug 04 18:31:55.648139 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:55.648120    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b9310caab6a4caefe5354cead99129721ad7efb514b7d51515ed4c8f0de926e\": container with ID starting with 9b9310caab6a4caefe5354cead99129721ad7efb514b7d51515ed4c8f0de926e not found: ID does not exist" containerID="9b9310caab6a4caefe5354cead99129721ad7efb514b7d51515ed4c8f0de926e"
Aug 04 18:31:55.648214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.648146    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b9310caab6a4caefe5354cead99129721ad7efb514b7d51515ed4c8f0de926e"} err="failed to get container status \"9b9310caab6a4caefe5354cead99129721ad7efb514b7d51515ed4c8f0de926e\": rpc error: code = NotFound desc = could not find container \"9b9310caab6a4caefe5354cead99129721ad7efb514b7d51515ed4c8f0de926e\": container with ID starting with 9b9310caab6a4caefe5354cead99129721ad7efb514b7d51515ed4c8f0de926e not found: ID does not exist"
Aug 04 18:31:55.648214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.648161    2566 scope.go:117] "RemoveContainer" containerID="7ab6500b2aebd0d64a26a4eebcd57347de0a9776150694de9e5fd9acd33b626e"
Aug 04 18:31:55.651099 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.651077    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"]
Aug 04 18:31:55.654122 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.654092    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-d6d6859cf-dq7l7"]
Aug 04 18:31:55.656590 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.656571    2566 scope.go:117] "RemoveContainer" containerID="d0f7f19eea841f0950708e39e3c30c5ad9d49f061b70a17b5886f598339662d6"
Aug 04 18:31:55.663971 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.663954    2566 scope.go:117] "RemoveContainer" containerID="d364e4a108af28fee081751ee9ae0757e3c4b1424e5eff93be3816f655fa3cd4"
Aug 04 18:31:55.668445 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.668421    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"]
Aug 04 18:31:55.671509 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.671487    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65d8dbfcdb-xcggl"]
Aug 04 18:31:55.676455 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.676432    2566 scope.go:117] "RemoveContainer" containerID="7ab6500b2aebd0d64a26a4eebcd57347de0a9776150694de9e5fd9acd33b626e"
Aug 04 18:31:55.676723 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:55.676704    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab6500b2aebd0d64a26a4eebcd57347de0a9776150694de9e5fd9acd33b626e\": container with ID starting with 7ab6500b2aebd0d64a26a4eebcd57347de0a9776150694de9e5fd9acd33b626e not found: ID does not exist" containerID="7ab6500b2aebd0d64a26a4eebcd57347de0a9776150694de9e5fd9acd33b626e"
Aug 04 18:31:55.676783 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.676733    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab6500b2aebd0d64a26a4eebcd57347de0a9776150694de9e5fd9acd33b626e"} err="failed to get container status \"7ab6500b2aebd0d64a26a4eebcd57347de0a9776150694de9e5fd9acd33b626e\": rpc error: code = NotFound desc = could not find container \"7ab6500b2aebd0d64a26a4eebcd57347de0a9776150694de9e5fd9acd33b626e\": container with ID starting with 7ab6500b2aebd0d64a26a4eebcd57347de0a9776150694de9e5fd9acd33b626e not found: ID does not exist"
Aug 04 18:31:55.676783 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.676753    2566 scope.go:117] "RemoveContainer" containerID="d0f7f19eea841f0950708e39e3c30c5ad9d49f061b70a17b5886f598339662d6"
Aug 04 18:31:55.676984 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:55.676971    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f7f19eea841f0950708e39e3c30c5ad9d49f061b70a17b5886f598339662d6\": container with ID starting with d0f7f19eea841f0950708e39e3c30c5ad9d49f061b70a17b5886f598339662d6 not found: ID does not exist" containerID="d0f7f19eea841f0950708e39e3c30c5ad9d49f061b70a17b5886f598339662d6"
Aug 04 18:31:55.677032 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.676986    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f7f19eea841f0950708e39e3c30c5ad9d49f061b70a17b5886f598339662d6"} err="failed to get container status \"d0f7f19eea841f0950708e39e3c30c5ad9d49f061b70a17b5886f598339662d6\": rpc error: code = NotFound desc = could not find container \"d0f7f19eea841f0950708e39e3c30c5ad9d49f061b70a17b5886f598339662d6\": container with ID starting with d0f7f19eea841f0950708e39e3c30c5ad9d49f061b70a17b5886f598339662d6 not found: ID does not exist"
Aug 04 18:31:55.677032 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.676999    2566 scope.go:117] "RemoveContainer" containerID="d364e4a108af28fee081751ee9ae0757e3c4b1424e5eff93be3816f655fa3cd4"
Aug 04 18:31:55.677223 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:31:55.677211    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d364e4a108af28fee081751ee9ae0757e3c4b1424e5eff93be3816f655fa3cd4\": container with ID starting with d364e4a108af28fee081751ee9ae0757e3c4b1424e5eff93be3816f655fa3cd4 not found: ID does not exist" containerID="d364e4a108af28fee081751ee9ae0757e3c4b1424e5eff93be3816f655fa3cd4"
Aug 04 18:31:55.677264 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.677225    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d364e4a108af28fee081751ee9ae0757e3c4b1424e5eff93be3816f655fa3cd4"} err="failed to get container status \"d364e4a108af28fee081751ee9ae0757e3c4b1424e5eff93be3816f655fa3cd4\": rpc error: code = NotFound desc = could not find container \"d364e4a108af28fee081751ee9ae0757e3c4b1424e5eff93be3816f655fa3cd4\": container with ID starting with d364e4a108af28fee081751ee9ae0757e3c4b1424e5eff93be3816f655fa3cd4 not found: ID does not exist"
Aug 04 18:31:55.709878 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.709847    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:31:55.851031 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.851005    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"]
Aug 04 18:31:55.853453 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:31:55.853425    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a67838d_68a3_4750_b04d_282d5c375e15.slice/crio-b47940f1548e6367d961c25ad2247ad8a74e5ab369b13d07148c3c6c4488ae3d WatchSource:0}: Error finding container b47940f1548e6367d961c25ad2247ad8a74e5ab369b13d07148c3c6c4488ae3d: Status 404 returned error can't find the container with id b47940f1548e6367d961c25ad2247ad8a74e5ab369b13d07148c3c6c4488ae3d
Aug 04 18:31:55.993473 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.993441    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-proxy-tls\") pod \"isvc-canary-promote-predictor-79fd8cbff8-2kc5w\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:55.995817 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:55.995789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-proxy-tls\") pod \"isvc-canary-promote-predictor-79fd8cbff8-2kc5w\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:56.285723 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:56.285680    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:31:56.427059 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:56.427032    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"]
Aug 04 18:31:56.428665 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:31:56.428637    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e7f1f4_8004_4f57_907a_9b04e2af2a6e.slice/crio-45ce2e6dc3f8ff4ea983653468b88f5c97c976cde8a418747d929bd335a83674 WatchSource:0}: Error finding container 45ce2e6dc3f8ff4ea983653468b88f5c97c976cde8a418747d929bd335a83674: Status 404 returned error can't find the container with id 45ce2e6dc3f8ff4ea983653468b88f5c97c976cde8a418747d929bd335a83674
Aug 04 18:31:56.629057 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:56.629019    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" event={"ID":"38e7f1f4-8004-4f57-907a-9b04e2af2a6e","Type":"ContainerStarted","Data":"1853f50c6770893805f8f416fac46c891eb5c363c595572e4d13a1510b85d3c1"}
Aug 04 18:31:56.629586 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:56.629079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" event={"ID":"38e7f1f4-8004-4f57-907a-9b04e2af2a6e","Type":"ContainerStarted","Data":"45ce2e6dc3f8ff4ea983653468b88f5c97c976cde8a418747d929bd335a83674"}
Aug 04 18:31:56.631454 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:56.631411    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" event={"ID":"3a67838d-68a3-4750-b04d-282d5c375e15","Type":"ContainerStarted","Data":"ef8f0fb3581e22759495499f7ccb4f24c7bcce0535e7df22c583a6938e8b431d"}
Aug 04 18:31:56.631571 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:56.631474    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" event={"ID":"3a67838d-68a3-4750-b04d-282d5c375e15","Type":"ContainerStarted","Data":"b47940f1548e6367d961c25ad2247ad8a74e5ab369b13d07148c3c6c4488ae3d"}
Aug 04 18:31:57.194207 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:57.194175    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00ea82f3-57c6-4de8-a960-37f114fb4170" path="/var/lib/kubelet/pods/00ea82f3-57c6-4de8-a960-37f114fb4170/volumes"
Aug 04 18:31:57.194716 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:57.194702    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f" path="/var/lib/kubelet/pods/67f5e3d2-53a1-4a3b-bd92-e4c0eaf2491f/volumes"
Aug 04 18:31:59.649287 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:59.649255    2566 generic.go:356] "Generic (PLEG): container finished" podID="3a67838d-68a3-4750-b04d-282d5c375e15" containerID="ef8f0fb3581e22759495499f7ccb4f24c7bcce0535e7df22c583a6938e8b431d" exitCode=0
Aug 04 18:31:59.649758 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:31:59.649294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" event={"ID":"3a67838d-68a3-4750-b04d-282d5c375e15","Type":"ContainerDied","Data":"ef8f0fb3581e22759495499f7ccb4f24c7bcce0535e7df22c583a6938e8b431d"}
Aug 04 18:32:00.655674 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:00.655638    2566 generic.go:356] "Generic (PLEG): container finished" podID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerID="1853f50c6770893805f8f416fac46c891eb5c363c595572e4d13a1510b85d3c1" exitCode=0
Aug 04 18:32:00.656079 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:00.655712    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" event={"ID":"38e7f1f4-8004-4f57-907a-9b04e2af2a6e","Type":"ContainerDied","Data":"1853f50c6770893805f8f416fac46c891eb5c363c595572e4d13a1510b85d3c1"}
Aug 04 18:32:00.657943 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:00.657920    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" event={"ID":"3a67838d-68a3-4750-b04d-282d5c375e15","Type":"ContainerStarted","Data":"c0aeac4fbdba1d589139d88a04c6b0ad4e14da091304e7586d9e140c925f909b"}
Aug 04 18:32:00.658056 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:00.657982    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" event={"ID":"3a67838d-68a3-4750-b04d-282d5c375e15","Type":"ContainerStarted","Data":"fdf18a1ce30d7e95d7bf505c86fe9d83ced3c52062e579df0346dbd06e322663"}
Aug 04 18:32:00.658318 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:00.658297    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:32:00.658407 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:00.658371    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:32:00.659584 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:00.659562    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Aug 04 18:32:00.697122 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:00.697049    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" podStartSLOduration=2.042878459 podStartE2EDuration="5.697026719s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.65414826s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:32:00.696644633 +0000 UTC m=+1919.987258833" watchObservedRunningTime="2026-08-04 18:32:00.697026719 +0000 UTC m=+1919.987640893"
Aug 04 18:32:01.665227 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:01.665195    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" event={"ID":"38e7f1f4-8004-4f57-907a-9b04e2af2a6e","Type":"ContainerStarted","Data":"a30492cbd30d44ebf68df2d1eac18c97b77b6f56b16c0158138e8b35eec97b82"}
Aug 04 18:32:01.665629 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:01.665252    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" event={"ID":"38e7f1f4-8004-4f57-907a-9b04e2af2a6e","Type":"ContainerStarted","Data":"c10640c13280e4b910e736dfcc06d2b469b9fc6e5ebdb0258757e742414eed8a"}
Aug 04 18:32:01.665629 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:01.665465    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Aug 04 18:32:01.665744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:01.665656    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:32:01.665858 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:01.665806    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:32:01.667036 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:01.667010    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Aug 04 18:32:01.687930 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:01.687836    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" podStartSLOduration=3.10446219 podStartE2EDuration="6.687820564s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.583358374s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:32:01.685012887 +0000 UTC m=+1920.975627079" watchObservedRunningTime="2026-08-04 18:32:01.687820564 +0000 UTC m=+1920.978434734"
Aug 04 18:32:02.670233 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:02.670197    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Aug 04 18:32:06.670084 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:06.670054    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:32:06.670703 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:06.670676    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Aug 04 18:32:07.674371 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:07.674335    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:32:07.674838 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:07.674811    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Aug 04 18:32:16.671490 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:16.671451    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Aug 04 18:32:17.675602 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:17.675563    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Aug 04 18:32:24.647441 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:24.647390    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6" podUID="5754703d-ce56-4877-ad5c-b26eb4ce32ee" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Aug 04 18:32:26.670674 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:26.670638    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Aug 04 18:32:27.674866 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:27.674824    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Aug 04 18:32:34.595130 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:34.595090    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:32:36.671105 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:36.671063    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Aug 04 18:32:37.674991 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:37.674956    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Aug 04 18:32:41.311481 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:41.311437    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"]
Aug 04 18:32:41.311884 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:41.311701    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6" podUID="5754703d-ce56-4877-ad5c-b26eb4ce32ee" containerName="kube-rbac-proxy" containerID="cri-o://e24f41a16e6c27d8831cf911c45ede25292edd1e5f87af2bfb3db3dd660eec0b" gracePeriod=30
Aug 04 18:32:41.311884 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:41.311701    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6" podUID="5754703d-ce56-4877-ad5c-b26eb4ce32ee" containerName="kserve-container" containerID="cri-o://bae3aa4c926ef0863f76556c00958010eedf5d3a11a3f811cdc9bfadb6e80e9a" gracePeriod=30
Aug 04 18:32:41.862680 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:41.862650    2566 generic.go:356] "Generic (PLEG): container finished" podID="5754703d-ce56-4877-ad5c-b26eb4ce32ee" containerID="e24f41a16e6c27d8831cf911c45ede25292edd1e5f87af2bfb3db3dd660eec0b" exitCode=2
Aug 04 18:32:41.862847 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:41.862725    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6" event={"ID":"5754703d-ce56-4877-ad5c-b26eb4ce32ee","Type":"ContainerDied","Data":"e24f41a16e6c27d8831cf911c45ede25292edd1e5f87af2bfb3db3dd660eec0b"}
Aug 04 18:32:44.586373 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:44.586323    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6" podUID="5754703d-ce56-4877-ad5c-b26eb4ce32ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Aug 04 18:32:46.532728 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.532693    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"]
Aug 04 18:32:46.536403 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.536385    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:46.539676 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.539656    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Aug 04 18:32:46.539750 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.539680    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Aug 04 18:32:46.549384 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.549334    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"]
Aug 04 18:32:46.549815 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.549781    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:46.549960 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.549829    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:46.549960 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.549869    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:46.550088 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.549957    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:46.550088 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.550060    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbm4c\" (UniqueName: \"kubernetes.io/projected/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kube-api-access-fbm4c\") pod \"isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:46.651130 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.651095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:46.651312 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.651193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbm4c\" (UniqueName: \"kubernetes.io/projected/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kube-api-access-fbm4c\") pod \"isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:46.651312 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.651281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:46.651312 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.651306    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:46.651525 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.651343    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:46.651767 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.651740    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:46.652117 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.652083    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:46.652117 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.652104    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:46.654084 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.654054    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:46.660903 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.660885    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbm4c\" (UniqueName: \"kubernetes.io/projected/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kube-api-access-fbm4c\") pod \"isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:46.671486 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.671455    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Aug 04 18:32:46.848693 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:46.848643    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:47.004151 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:47.004087    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"]
Aug 04 18:32:47.006617 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:32:47.006578    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca193a8_5e10_4901_a0e8_6b5b22a2197c.slice/crio-85aeb4f036970202246bdf60eb4797964ee3ffcb27a85b8a9b5cb4951eed5f89 WatchSource:0}: Error finding container 85aeb4f036970202246bdf60eb4797964ee3ffcb27a85b8a9b5cb4951eed5f89: Status 404 returned error can't find the container with id 85aeb4f036970202246bdf60eb4797964ee3ffcb27a85b8a9b5cb4951eed5f89
Aug 04 18:32:47.674790 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:47.674751    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Aug 04 18:32:47.892739 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:47.892705    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" event={"ID":"6ca193a8-5e10-4901-a0e8-6b5b22a2197c","Type":"ContainerStarted","Data":"cbcc7736db6347389e21d3a15ccac0ed01cf8dc304d761264be9734f8446417c"}
Aug 04 18:32:47.892920 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:47.892764    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" event={"ID":"6ca193a8-5e10-4901-a0e8-6b5b22a2197c","Type":"ContainerStarted","Data":"85aeb4f036970202246bdf60eb4797964ee3ffcb27a85b8a9b5cb4951eed5f89"}
Aug 04 18:32:49.460625 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.460602    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:32:49.479648 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.479616    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kube-rbac-proxy-sar-config\") pod \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") "
Aug 04 18:32:49.479648 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.479651    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5754703d-ce56-4877-ad5c-b26eb4ce32ee-proxy-tls\") pod \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") "
Aug 04 18:32:49.479855 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.479681    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx2km\" (UniqueName: \"kubernetes.io/projected/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kube-api-access-sx2km\") pod \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") "
Aug 04 18:32:49.479855 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.479790    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kserve-provision-location\") pod \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") "
Aug 04 18:32:49.479855 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.479841    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5754703d-ce56-4877-ad5c-b26eb4ce32ee-cabundle-cert\") pod \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\" (UID: \"5754703d-ce56-4877-ad5c-b26eb4ce32ee\") "
Aug 04 18:32:49.480050 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.480016    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5754703d-ce56-4877-ad5c-b26eb4ce32ee" (UID: "5754703d-ce56-4877-ad5c-b26eb4ce32ee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:32:49.480141 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.480105    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5754703d-ce56-4877-ad5c-b26eb4ce32ee" (UID: "5754703d-ce56-4877-ad5c-b26eb4ce32ee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:32:49.480201 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.480154    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5754703d-ce56-4877-ad5c-b26eb4ce32ee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5754703d-ce56-4877-ad5c-b26eb4ce32ee" (UID: "5754703d-ce56-4877-ad5c-b26eb4ce32ee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:32:49.481740 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.481716    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5754703d-ce56-4877-ad5c-b26eb4ce32ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5754703d-ce56-4877-ad5c-b26eb4ce32ee" (UID: "5754703d-ce56-4877-ad5c-b26eb4ce32ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:32:49.481848 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.481747    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kube-api-access-sx2km" (OuterVolumeSpecName: "kube-api-access-sx2km") pod "5754703d-ce56-4877-ad5c-b26eb4ce32ee" (UID: "5754703d-ce56-4877-ad5c-b26eb4ce32ee"). InnerVolumeSpecName "kube-api-access-sx2km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:32:49.581007 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.580972    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5754703d-ce56-4877-ad5c-b26eb4ce32ee-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:32:49.581007 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.580999    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx2km\" (UniqueName: \"kubernetes.io/projected/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kube-api-access-sx2km\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:32:49.581007 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.581008    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:32:49.581007 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.581018    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5754703d-ce56-4877-ad5c-b26eb4ce32ee-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:32:49.581286 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.581027    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5754703d-ce56-4877-ad5c-b26eb4ce32ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:32:49.905421 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.905310    2566 generic.go:356] "Generic (PLEG): container finished" podID="5754703d-ce56-4877-ad5c-b26eb4ce32ee" containerID="bae3aa4c926ef0863f76556c00958010eedf5d3a11a3f811cdc9bfadb6e80e9a" exitCode=0
Aug 04 18:32:49.905421 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.905382    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6" event={"ID":"5754703d-ce56-4877-ad5c-b26eb4ce32ee","Type":"ContainerDied","Data":"bae3aa4c926ef0863f76556c00958010eedf5d3a11a3f811cdc9bfadb6e80e9a"}
Aug 04 18:32:49.905421 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.905401    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"
Aug 04 18:32:49.905695 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.905444    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6" event={"ID":"5754703d-ce56-4877-ad5c-b26eb4ce32ee","Type":"ContainerDied","Data":"78c6b80ef11b623c81da292f1cc6b8ef617a580a3b79adda25c0e06c74c1df54"}
Aug 04 18:32:49.905695 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.905471    2566 scope.go:117] "RemoveContainer" containerID="e24f41a16e6c27d8831cf911c45ede25292edd1e5f87af2bfb3db3dd660eec0b"
Aug 04 18:32:49.914600 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.914584    2566 scope.go:117] "RemoveContainer" containerID="bae3aa4c926ef0863f76556c00958010eedf5d3a11a3f811cdc9bfadb6e80e9a"
Aug 04 18:32:49.922604 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.922583    2566 scope.go:117] "RemoveContainer" containerID="358d4807d8935d32fa54b7d965dcdc504c411b6d4e8f19ce3a6b09a52d5fce73"
Aug 04 18:32:49.930410 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.930390    2566 scope.go:117] "RemoveContainer" containerID="e24f41a16e6c27d8831cf911c45ede25292edd1e5f87af2bfb3db3dd660eec0b"
Aug 04 18:32:49.930709 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:32:49.930682    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24f41a16e6c27d8831cf911c45ede25292edd1e5f87af2bfb3db3dd660eec0b\": container with ID starting with e24f41a16e6c27d8831cf911c45ede25292edd1e5f87af2bfb3db3dd660eec0b not found: ID does not exist" containerID="e24f41a16e6c27d8831cf911c45ede25292edd1e5f87af2bfb3db3dd660eec0b"
Aug 04 18:32:49.930781 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.930723    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24f41a16e6c27d8831cf911c45ede25292edd1e5f87af2bfb3db3dd660eec0b"} err="failed to get container status \"e24f41a16e6c27d8831cf911c45ede25292edd1e5f87af2bfb3db3dd660eec0b\": rpc error: code = NotFound desc = could not find container \"e24f41a16e6c27d8831cf911c45ede25292edd1e5f87af2bfb3db3dd660eec0b\": container with ID starting with e24f41a16e6c27d8831cf911c45ede25292edd1e5f87af2bfb3db3dd660eec0b not found: ID does not exist"
Aug 04 18:32:49.930781 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.930748    2566 scope.go:117] "RemoveContainer" containerID="bae3aa4c926ef0863f76556c00958010eedf5d3a11a3f811cdc9bfadb6e80e9a"
Aug 04 18:32:49.931009 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:32:49.930980    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae3aa4c926ef0863f76556c00958010eedf5d3a11a3f811cdc9bfadb6e80e9a\": container with ID starting with bae3aa4c926ef0863f76556c00958010eedf5d3a11a3f811cdc9bfadb6e80e9a not found: ID does not exist" containerID="bae3aa4c926ef0863f76556c00958010eedf5d3a11a3f811cdc9bfadb6e80e9a"
Aug 04 18:32:49.931115 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.931016    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae3aa4c926ef0863f76556c00958010eedf5d3a11a3f811cdc9bfadb6e80e9a"} err="failed to get container status \"bae3aa4c926ef0863f76556c00958010eedf5d3a11a3f811cdc9bfadb6e80e9a\": rpc error: code = NotFound desc = could not find container \"bae3aa4c926ef0863f76556c00958010eedf5d3a11a3f811cdc9bfadb6e80e9a\": container with ID starting with bae3aa4c926ef0863f76556c00958010eedf5d3a11a3f811cdc9bfadb6e80e9a not found: ID does not exist"
Aug 04 18:32:49.931115 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.931038    2566 scope.go:117] "RemoveContainer" containerID="358d4807d8935d32fa54b7d965dcdc504c411b6d4e8f19ce3a6b09a52d5fce73"
Aug 04 18:32:49.931530 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:32:49.931476    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358d4807d8935d32fa54b7d965dcdc504c411b6d4e8f19ce3a6b09a52d5fce73\": container with ID starting with 358d4807d8935d32fa54b7d965dcdc504c411b6d4e8f19ce3a6b09a52d5fce73 not found: ID does not exist" containerID="358d4807d8935d32fa54b7d965dcdc504c411b6d4e8f19ce3a6b09a52d5fce73"
Aug 04 18:32:49.931530 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.931507    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358d4807d8935d32fa54b7d965dcdc504c411b6d4e8f19ce3a6b09a52d5fce73"} err="failed to get container status \"358d4807d8935d32fa54b7d965dcdc504c411b6d4e8f19ce3a6b09a52d5fce73\": rpc error: code = NotFound desc = could not find container \"358d4807d8935d32fa54b7d965dcdc504c411b6d4e8f19ce3a6b09a52d5fce73\": container with ID starting with 358d4807d8935d32fa54b7d965dcdc504c411b6d4e8f19ce3a6b09a52d5fce73 not found: ID does not exist"
Aug 04 18:32:49.933987 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.933963    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"]
Aug 04 18:32:49.936661 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:49.936641    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-bwlb6"]
Aug 04 18:32:50.912952 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:50.912920    2566 generic.go:356] "Generic (PLEG): container finished" podID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerID="cbcc7736db6347389e21d3a15ccac0ed01cf8dc304d761264be9734f8446417c" exitCode=0
Aug 04 18:32:50.913312 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:50.912991    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" event={"ID":"6ca193a8-5e10-4901-a0e8-6b5b22a2197c","Type":"ContainerDied","Data":"cbcc7736db6347389e21d3a15ccac0ed01cf8dc304d761264be9734f8446417c"}
Aug 04 18:32:51.194785 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:51.194710    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5754703d-ce56-4877-ad5c-b26eb4ce32ee" path="/var/lib/kubelet/pods/5754703d-ce56-4877-ad5c-b26eb4ce32ee/volumes"
Aug 04 18:32:51.920611 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:51.920575    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" event={"ID":"6ca193a8-5e10-4901-a0e8-6b5b22a2197c","Type":"ContainerStarted","Data":"3ddd50a1b2aff6a0e4aa519c8cfd522e72b62c2cb3f608186063e500b0495090"}
Aug 04 18:32:51.921013 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:51.920639    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" event={"ID":"6ca193a8-5e10-4901-a0e8-6b5b22a2197c","Type":"ContainerStarted","Data":"75875baf8e1d131507f512811416b3c3b60c3ddeeef42835ffc7f341705a3d18"}
Aug 04 18:32:51.921013 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:51.920829    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:51.942666 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:51.942610    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" podStartSLOduration=2.197058738 podStartE2EDuration="5.942591377s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.745532639s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:32:51.94074771 +0000 UTC m=+1971.231361899" watchObservedRunningTime="2026-08-04 18:32:51.942591377 +0000 UTC m=+1971.233205550"
Aug 04 18:32:52.926547 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:52.926513    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:52.927839 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:52.927810    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Aug 04 18:32:53.930788 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:53.930750    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Aug 04 18:32:56.671030 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:56.670992    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Aug 04 18:32:57.675828 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:57.675787    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Aug 04 18:32:58.935592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:58.935563    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:32:58.936148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:32:58.936120    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Aug 04 18:33:06.671344 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:06.671302    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Aug 04 18:33:07.675464 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:07.675430    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Aug 04 18:33:08.936181 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:08.936140    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Aug 04 18:33:15.189983 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:15.189934    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Aug 04 18:33:17.675497 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:17.675471    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:33:18.936578 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:18.936538    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Aug 04 18:33:25.195228 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:25.195200    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:33:25.472652 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:25.472572    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"]
Aug 04 18:33:25.472882 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:25.472839    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kserve-container" containerID="cri-o://c10640c13280e4b910e736dfcc06d2b469b9fc6e5ebdb0258757e742414eed8a" gracePeriod=30
Aug 04 18:33:25.472959 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:25.472890    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kube-rbac-proxy" containerID="cri-o://a30492cbd30d44ebf68df2d1eac18c97b77b6f56b16c0158138e8b35eec97b82" gracePeriod=30
Aug 04 18:33:26.087920 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:26.087887    2566 generic.go:356] "Generic (PLEG): container finished" podID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerID="a30492cbd30d44ebf68df2d1eac18c97b77b6f56b16c0158138e8b35eec97b82" exitCode=2
Aug 04 18:33:26.088076 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:26.087962    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" event={"ID":"38e7f1f4-8004-4f57-907a-9b04e2af2a6e","Type":"ContainerDied","Data":"a30492cbd30d44ebf68df2d1eac18c97b77b6f56b16c0158138e8b35eec97b82"}
Aug 04 18:33:27.671125 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:27.671084    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Aug 04 18:33:27.675445 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:27.675419    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Aug 04 18:33:28.936403 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:28.936338    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Aug 04 18:33:30.116672 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.111763    2566 generic.go:356] "Generic (PLEG): container finished" podID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerID="c10640c13280e4b910e736dfcc06d2b469b9fc6e5ebdb0258757e742414eed8a" exitCode=0
Aug 04 18:33:30.116672 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.111857    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" event={"ID":"38e7f1f4-8004-4f57-907a-9b04e2af2a6e","Type":"ContainerDied","Data":"c10640c13280e4b910e736dfcc06d2b469b9fc6e5ebdb0258757e742414eed8a"}
Aug 04 18:33:30.223684 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.223661    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:33:30.267238 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.267209    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-cabundle-cert\") pod \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") "
Aug 04 18:33:30.267419 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.267322    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kube-rbac-proxy-sar-config\") pod \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") "
Aug 04 18:33:30.267419 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.267347    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-proxy-tls\") pod \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") "
Aug 04 18:33:30.267613 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.267585    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bxfz\" (UniqueName: \"kubernetes.io/projected/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kube-api-access-8bxfz\") pod \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") "
Aug 04 18:33:30.267689 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.267636    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kserve-provision-location\") pod \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\" (UID: \"38e7f1f4-8004-4f57-907a-9b04e2af2a6e\") "
Aug 04 18:33:30.267689 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.267585    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "38e7f1f4-8004-4f57-907a-9b04e2af2a6e" (UID: "38e7f1f4-8004-4f57-907a-9b04e2af2a6e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:33:30.267804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.267651    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "38e7f1f4-8004-4f57-907a-9b04e2af2a6e" (UID: "38e7f1f4-8004-4f57-907a-9b04e2af2a6e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:33:30.267923 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.267907    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:33:30.267973 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.267927    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:33:30.268008 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.267971    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38e7f1f4-8004-4f57-907a-9b04e2af2a6e" (UID: "38e7f1f4-8004-4f57-907a-9b04e2af2a6e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:33:30.269558 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.269537    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "38e7f1f4-8004-4f57-907a-9b04e2af2a6e" (UID: "38e7f1f4-8004-4f57-907a-9b04e2af2a6e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:33:30.269725 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.269705    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kube-api-access-8bxfz" (OuterVolumeSpecName: "kube-api-access-8bxfz") pod "38e7f1f4-8004-4f57-907a-9b04e2af2a6e" (UID: "38e7f1f4-8004-4f57-907a-9b04e2af2a6e"). InnerVolumeSpecName "kube-api-access-8bxfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:33:30.368818 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.368733    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:33:30.368818 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.368762    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bxfz\" (UniqueName: \"kubernetes.io/projected/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kube-api-access-8bxfz\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:33:30.368818 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:30.368773    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38e7f1f4-8004-4f57-907a-9b04e2af2a6e-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:33:31.119555 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:31.119522    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w" event={"ID":"38e7f1f4-8004-4f57-907a-9b04e2af2a6e","Type":"ContainerDied","Data":"45ce2e6dc3f8ff4ea983653468b88f5c97c976cde8a418747d929bd335a83674"}
Aug 04 18:33:31.119966 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:31.119570    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"
Aug 04 18:33:31.119966 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:31.119590    2566 scope.go:117] "RemoveContainer" containerID="a30492cbd30d44ebf68df2d1eac18c97b77b6f56b16c0158138e8b35eec97b82"
Aug 04 18:33:31.128649 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:31.128630    2566 scope.go:117] "RemoveContainer" containerID="c10640c13280e4b910e736dfcc06d2b469b9fc6e5ebdb0258757e742414eed8a"
Aug 04 18:33:31.136584 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:31.136566    2566 scope.go:117] "RemoveContainer" containerID="1853f50c6770893805f8f416fac46c891eb5c363c595572e4d13a1510b85d3c1"
Aug 04 18:33:31.154728 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:31.154698    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"]
Aug 04 18:33:31.157448 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:31.157423    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-79fd8cbff8-2kc5w"]
Aug 04 18:33:31.194872 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:31.194842    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" path="/var/lib/kubelet/pods/38e7f1f4-8004-4f57-907a-9b04e2af2a6e/volumes"
Aug 04 18:33:38.936020 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:38.935983    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Aug 04 18:33:48.936740 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:48.936700    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Aug 04 18:33:58.936894 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:33:58.936856    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Aug 04 18:34:08.936519 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:08.936487    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:34:16.825094 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:16.825059    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"]
Aug 04 18:34:16.825942 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:16.825426    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kube-rbac-proxy" containerID="cri-o://3ddd50a1b2aff6a0e4aa519c8cfd522e72b62c2cb3f608186063e500b0495090" gracePeriod=30
Aug 04 18:34:16.825942 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:16.825426    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kserve-container" containerID="cri-o://75875baf8e1d131507f512811416b3c3b60c3ddeeef42835ffc7f341705a3d18" gracePeriod=30
Aug 04 18:34:17.343088 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:17.343055    2566 generic.go:356] "Generic (PLEG): container finished" podID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerID="3ddd50a1b2aff6a0e4aa519c8cfd522e72b62c2cb3f608186063e500b0495090" exitCode=2
Aug 04 18:34:17.343257 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:17.343128    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" event={"ID":"6ca193a8-5e10-4901-a0e8-6b5b22a2197c","Type":"ContainerDied","Data":"3ddd50a1b2aff6a0e4aa519c8cfd522e72b62c2cb3f608186063e500b0495090"}
Aug 04 18:34:18.931383 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:18.931319    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Aug 04 18:34:18.936679 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:18.936653    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Aug 04 18:34:21.577779 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:21.577752    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:34:21.627184 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:21.627110    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kserve-provision-location\") pod \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") "
Aug 04 18:34:21.627184 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:21.627181    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-proxy-tls\") pod \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") "
Aug 04 18:34:21.627434 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:21.627200    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbm4c\" (UniqueName: \"kubernetes.io/projected/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kube-api-access-fbm4c\") pod \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") "
Aug 04 18:34:21.627434 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:21.627232    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kube-rbac-proxy-sar-config\") pod \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") "
Aug 04 18:34:21.627434 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:21.627254    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-cabundle-cert\") pod \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\" (UID: \"6ca193a8-5e10-4901-a0e8-6b5b22a2197c\") "
Aug 04 18:34:21.627641 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:21.627525    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ca193a8-5e10-4901-a0e8-6b5b22a2197c" (UID: "6ca193a8-5e10-4901-a0e8-6b5b22a2197c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:34:21.627641 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:21.627611    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ca193a8-5e10-4901-a0e8-6b5b22a2197c" (UID: "6ca193a8-5e10-4901-a0e8-6b5b22a2197c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:34:21.627744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:21.627715    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6ca193a8-5e10-4901-a0e8-6b5b22a2197c" (UID: "6ca193a8-5e10-4901-a0e8-6b5b22a2197c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:34:21.629420 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:21.629395    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ca193a8-5e10-4901-a0e8-6b5b22a2197c" (UID: "6ca193a8-5e10-4901-a0e8-6b5b22a2197c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:34:21.629492 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:21.629458    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kube-api-access-fbm4c" (OuterVolumeSpecName: "kube-api-access-fbm4c") pod "6ca193a8-5e10-4901-a0e8-6b5b22a2197c" (UID: "6ca193a8-5e10-4901-a0e8-6b5b22a2197c"). InnerVolumeSpecName "kube-api-access-fbm4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:34:21.728669 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:21.728635    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:34:21.728669 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:21.728662    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbm4c\" (UniqueName: \"kubernetes.io/projected/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kube-api-access-fbm4c\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:34:21.728669 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:21.728673    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:34:21.728669 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:21.728682    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:34:21.728939 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:21.728691    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ca193a8-5e10-4901-a0e8-6b5b22a2197c-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:34:22.156756 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.156716    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"]
Aug 04 18:34:22.157167 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157149    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="storage-initializer"
Aug 04 18:34:22.157265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157170    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="storage-initializer"
Aug 04 18:34:22.157265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157183    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kserve-container"
Aug 04 18:34:22.157265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157190    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kserve-container"
Aug 04 18:34:22.157265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157202    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kube-rbac-proxy"
Aug 04 18:34:22.157265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157211    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kube-rbac-proxy"
Aug 04 18:34:22.157265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157221    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kserve-container"
Aug 04 18:34:22.157265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157230    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kserve-container"
Aug 04 18:34:22.157265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157244    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5754703d-ce56-4877-ad5c-b26eb4ce32ee" containerName="kube-rbac-proxy"
Aug 04 18:34:22.157265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157251    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5754703d-ce56-4877-ad5c-b26eb4ce32ee" containerName="kube-rbac-proxy"
Aug 04 18:34:22.157265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157263    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5754703d-ce56-4877-ad5c-b26eb4ce32ee" containerName="storage-initializer"
Aug 04 18:34:22.157812 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157271    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5754703d-ce56-4877-ad5c-b26eb4ce32ee" containerName="storage-initializer"
Aug 04 18:34:22.157812 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157281    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5754703d-ce56-4877-ad5c-b26eb4ce32ee" containerName="kserve-container"
Aug 04 18:34:22.157812 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157289    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5754703d-ce56-4877-ad5c-b26eb4ce32ee" containerName="kserve-container"
Aug 04 18:34:22.157812 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157301    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="storage-initializer"
Aug 04 18:34:22.157812 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157310    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="storage-initializer"
Aug 04 18:34:22.157812 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157334    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kube-rbac-proxy"
Aug 04 18:34:22.157812 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157343    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kube-rbac-proxy"
Aug 04 18:34:22.157812 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157441    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kserve-container"
Aug 04 18:34:22.157812 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157455    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerName="kube-rbac-proxy"
Aug 04 18:34:22.157812 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157464    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5754703d-ce56-4877-ad5c-b26eb4ce32ee" containerName="kserve-container"
Aug 04 18:34:22.157812 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157475    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kube-rbac-proxy"
Aug 04 18:34:22.157812 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157487    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5754703d-ce56-4877-ad5c-b26eb4ce32ee" containerName="kube-rbac-proxy"
Aug 04 18:34:22.157812 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.157496    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="38e7f1f4-8004-4f57-907a-9b04e2af2a6e" containerName="kserve-container"
Aug 04 18:34:22.161530 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.161506    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.170899 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.170874    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Aug 04 18:34:22.171030 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.170901    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Aug 04 18:34:22.185243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.185214    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"]
Aug 04 18:34:22.233883 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.233849    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/155b1987-b73f-46f4-953f-6b8fdc2a0f51-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.233883 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.233884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.234088 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.233936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.234088 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.233964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w69k8\" (UniqueName: \"kubernetes.io/projected/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kube-api-access-w69k8\") pod \"isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.234088 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.233994    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/155b1987-b73f-46f4-953f-6b8fdc2a0f51-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.335482 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.335443    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/155b1987-b73f-46f4-953f-6b8fdc2a0f51-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.335482 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.335482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.335831 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.335535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.335831 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.335569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w69k8\" (UniqueName: \"kubernetes.io/projected/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kube-api-access-w69k8\") pod \"isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.335831 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.335613    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/155b1987-b73f-46f4-953f-6b8fdc2a0f51-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.336041 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.336014    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.336322 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.336294    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/155b1987-b73f-46f4-953f-6b8fdc2a0f51-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.336391 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.336298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.338043 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.338022    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/155b1987-b73f-46f4-953f-6b8fdc2a0f51-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.347546 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.347524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w69k8\" (UniqueName: \"kubernetes.io/projected/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kube-api-access-w69k8\") pod \"isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.369662 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.369635    2566 generic.go:356] "Generic (PLEG): container finished" podID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" containerID="75875baf8e1d131507f512811416b3c3b60c3ddeeef42835ffc7f341705a3d18" exitCode=0
Aug 04 18:34:22.369793 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.369693    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"
Aug 04 18:34:22.369793 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.369712    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" event={"ID":"6ca193a8-5e10-4901-a0e8-6b5b22a2197c","Type":"ContainerDied","Data":"75875baf8e1d131507f512811416b3c3b60c3ddeeef42835ffc7f341705a3d18"}
Aug 04 18:34:22.369793 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.369783    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n" event={"ID":"6ca193a8-5e10-4901-a0e8-6b5b22a2197c","Type":"ContainerDied","Data":"85aeb4f036970202246bdf60eb4797964ee3ffcb27a85b8a9b5cb4951eed5f89"}
Aug 04 18:34:22.369900 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.369822    2566 scope.go:117] "RemoveContainer" containerID="3ddd50a1b2aff6a0e4aa519c8cfd522e72b62c2cb3f608186063e500b0495090"
Aug 04 18:34:22.379268 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.379253    2566 scope.go:117] "RemoveContainer" containerID="75875baf8e1d131507f512811416b3c3b60c3ddeeef42835ffc7f341705a3d18"
Aug 04 18:34:22.387162 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.387146    2566 scope.go:117] "RemoveContainer" containerID="cbcc7736db6347389e21d3a15ccac0ed01cf8dc304d761264be9734f8446417c"
Aug 04 18:34:22.394518 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.394496    2566 scope.go:117] "RemoveContainer" containerID="3ddd50a1b2aff6a0e4aa519c8cfd522e72b62c2cb3f608186063e500b0495090"
Aug 04 18:34:22.394767 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:34:22.394745    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ddd50a1b2aff6a0e4aa519c8cfd522e72b62c2cb3f608186063e500b0495090\": container with ID starting with 3ddd50a1b2aff6a0e4aa519c8cfd522e72b62c2cb3f608186063e500b0495090 not found: ID does not exist" containerID="3ddd50a1b2aff6a0e4aa519c8cfd522e72b62c2cb3f608186063e500b0495090"
Aug 04 18:34:22.394821 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.394776    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ddd50a1b2aff6a0e4aa519c8cfd522e72b62c2cb3f608186063e500b0495090"} err="failed to get container status \"3ddd50a1b2aff6a0e4aa519c8cfd522e72b62c2cb3f608186063e500b0495090\": rpc error: code = NotFound desc = could not find container \"3ddd50a1b2aff6a0e4aa519c8cfd522e72b62c2cb3f608186063e500b0495090\": container with ID starting with 3ddd50a1b2aff6a0e4aa519c8cfd522e72b62c2cb3f608186063e500b0495090 not found: ID does not exist"
Aug 04 18:34:22.394821 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.394794    2566 scope.go:117] "RemoveContainer" containerID="75875baf8e1d131507f512811416b3c3b60c3ddeeef42835ffc7f341705a3d18"
Aug 04 18:34:22.395016 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:34:22.394999    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75875baf8e1d131507f512811416b3c3b60c3ddeeef42835ffc7f341705a3d18\": container with ID starting with 75875baf8e1d131507f512811416b3c3b60c3ddeeef42835ffc7f341705a3d18 not found: ID does not exist" containerID="75875baf8e1d131507f512811416b3c3b60c3ddeeef42835ffc7f341705a3d18"
Aug 04 18:34:22.395060 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.395022    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75875baf8e1d131507f512811416b3c3b60c3ddeeef42835ffc7f341705a3d18"} err="failed to get container status \"75875baf8e1d131507f512811416b3c3b60c3ddeeef42835ffc7f341705a3d18\": rpc error: code = NotFound desc = could not find container \"75875baf8e1d131507f512811416b3c3b60c3ddeeef42835ffc7f341705a3d18\": container with ID starting with 75875baf8e1d131507f512811416b3c3b60c3ddeeef42835ffc7f341705a3d18 not found: ID does not exist"
Aug 04 18:34:22.395060 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.395043    2566 scope.go:117] "RemoveContainer" containerID="cbcc7736db6347389e21d3a15ccac0ed01cf8dc304d761264be9734f8446417c"
Aug 04 18:34:22.395286 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:34:22.395271    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbcc7736db6347389e21d3a15ccac0ed01cf8dc304d761264be9734f8446417c\": container with ID starting with cbcc7736db6347389e21d3a15ccac0ed01cf8dc304d761264be9734f8446417c not found: ID does not exist" containerID="cbcc7736db6347389e21d3a15ccac0ed01cf8dc304d761264be9734f8446417c"
Aug 04 18:34:22.395341 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.395293    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbcc7736db6347389e21d3a15ccac0ed01cf8dc304d761264be9734f8446417c"} err="failed to get container status \"cbcc7736db6347389e21d3a15ccac0ed01cf8dc304d761264be9734f8446417c\": rpc error: code = NotFound desc = could not find container \"cbcc7736db6347389e21d3a15ccac0ed01cf8dc304d761264be9734f8446417c\": container with ID starting with cbcc7736db6347389e21d3a15ccac0ed01cf8dc304d761264be9734f8446417c not found: ID does not exist"
Aug 04 18:34:22.409679 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.409622    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"]
Aug 04 18:34:22.414451 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.414429    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5dbb8d5f8c-vp58n"]
Aug 04 18:34:22.472459 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.472431    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:22.613056 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:22.613035    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"]
Aug 04 18:34:22.615247 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:34:22.615219    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod155b1987_b73f_46f4_953f_6b8fdc2a0f51.slice/crio-b7bd346cecce48fbd4d5b088bbf8e205b40a5cc9b946315f380e1fb702005338 WatchSource:0}: Error finding container b7bd346cecce48fbd4d5b088bbf8e205b40a5cc9b946315f380e1fb702005338: Status 404 returned error can't find the container with id b7bd346cecce48fbd4d5b088bbf8e205b40a5cc9b946315f380e1fb702005338
Aug 04 18:34:23.193812 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:23.193738    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ca193a8-5e10-4901-a0e8-6b5b22a2197c" path="/var/lib/kubelet/pods/6ca193a8-5e10-4901-a0e8-6b5b22a2197c/volumes"
Aug 04 18:34:23.377786 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:23.377747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" event={"ID":"155b1987-b73f-46f4-953f-6b8fdc2a0f51","Type":"ContainerStarted","Data":"9fc4a891f6d16bd6d8a888bdb4ba26bab58a733ba677e9171342d18d5033e4ae"}
Aug 04 18:34:23.377983 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:23.377814    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" event={"ID":"155b1987-b73f-46f4-953f-6b8fdc2a0f51","Type":"ContainerStarted","Data":"b7bd346cecce48fbd4d5b088bbf8e205b40a5cc9b946315f380e1fb702005338"}
Aug 04 18:34:26.392953 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:26.392921    2566 generic.go:356] "Generic (PLEG): container finished" podID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerID="9fc4a891f6d16bd6d8a888bdb4ba26bab58a733ba677e9171342d18d5033e4ae" exitCode=0
Aug 04 18:34:26.393328 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:26.393000    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" event={"ID":"155b1987-b73f-46f4-953f-6b8fdc2a0f51","Type":"ContainerDied","Data":"9fc4a891f6d16bd6d8a888bdb4ba26bab58a733ba677e9171342d18d5033e4ae"}
Aug 04 18:34:27.400753 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:27.400720    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" event={"ID":"155b1987-b73f-46f4-953f-6b8fdc2a0f51","Type":"ContainerStarted","Data":"b7695caf811398fe8d61a2c9ae0c533c9269b5d0355e9b474104f70a4ed7f809"}
Aug 04 18:34:27.401135 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:27.400781    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" event={"ID":"155b1987-b73f-46f4-953f-6b8fdc2a0f51","Type":"ContainerStarted","Data":"d1199441018912c99ea71a4874a38ecec9b716396434e760c8a23ad95672692a"}
Aug 04 18:34:27.401135 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:27.400986    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:27.428862 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:27.428811    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" podStartSLOduration=1.822040371 podStartE2EDuration="5.428794816s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.606754445s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:34:27.427995952 +0000 UTC m=+2066.718610166" watchObservedRunningTime="2026-08-04 18:34:27.428794816 +0000 UTC m=+2066.719408986"
Aug 04 18:34:28.405927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:28.405895    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:28.407412 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:28.407383    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Aug 04 18:34:29.410341 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:29.410302    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Aug 04 18:34:34.415634 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:34.415607    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:34:34.416126 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:34.416083    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Aug 04 18:34:44.416377 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:44.416321    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Aug 04 18:34:54.416844 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:34:54.416801    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Aug 04 18:35:04.416743 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:04.416701    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Aug 04 18:35:05.755873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:05.755835    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"]
Aug 04 18:35:05.756384 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:05.756134    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kserve-container" containerID="cri-o://fdf18a1ce30d7e95d7bf505c86fe9d83ced3c52062e579df0346dbd06e322663" gracePeriod=30
Aug 04 18:35:05.756384 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:05.756161    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kube-rbac-proxy" containerID="cri-o://c0aeac4fbdba1d589139d88a04c6b0ad4e14da091304e7586d9e140c925f909b" gracePeriod=30
Aug 04 18:35:06.592034 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:06.591998    2566 generic.go:356] "Generic (PLEG): container finished" podID="3a67838d-68a3-4750-b04d-282d5c375e15" containerID="c0aeac4fbdba1d589139d88a04c6b0ad4e14da091304e7586d9e140c925f909b" exitCode=2
Aug 04 18:35:06.592218 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:06.592067    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" event={"ID":"3a67838d-68a3-4750-b04d-282d5c375e15","Type":"ContainerDied","Data":"c0aeac4fbdba1d589139d88a04c6b0ad4e14da091304e7586d9e140c925f909b"}
Aug 04 18:35:06.666506 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:06.666461    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Aug 04 18:35:10.611844 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.611809    2566 generic.go:356] "Generic (PLEG): container finished" podID="3a67838d-68a3-4750-b04d-282d5c375e15" containerID="fdf18a1ce30d7e95d7bf505c86fe9d83ced3c52062e579df0346dbd06e322663" exitCode=0
Aug 04 18:35:10.612170 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.611878    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" event={"ID":"3a67838d-68a3-4750-b04d-282d5c375e15","Type":"ContainerDied","Data":"fdf18a1ce30d7e95d7bf505c86fe9d83ced3c52062e579df0346dbd06e322663"}
Aug 04 18:35:10.612170 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.611937    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss" event={"ID":"3a67838d-68a3-4750-b04d-282d5c375e15","Type":"ContainerDied","Data":"b47940f1548e6367d961c25ad2247ad8a74e5ab369b13d07148c3c6c4488ae3d"}
Aug 04 18:35:10.612170 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.611957    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b47940f1548e6367d961c25ad2247ad8a74e5ab369b13d07148c3c6c4488ae3d"
Aug 04 18:35:10.612366 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.612325    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:35:10.669120 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.669091    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a67838d-68a3-4750-b04d-282d5c375e15-proxy-tls\") pod \"3a67838d-68a3-4750-b04d-282d5c375e15\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") "
Aug 04 18:35:10.669308 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.669135    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a67838d-68a3-4750-b04d-282d5c375e15-kube-rbac-proxy-sar-config\") pod \"3a67838d-68a3-4750-b04d-282d5c375e15\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") "
Aug 04 18:35:10.669308 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.669165    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a67838d-68a3-4750-b04d-282d5c375e15-kserve-provision-location\") pod \"3a67838d-68a3-4750-b04d-282d5c375e15\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") "
Aug 04 18:35:10.669308 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.669187    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a67838d-68a3-4750-b04d-282d5c375e15-cabundle-cert\") pod \"3a67838d-68a3-4750-b04d-282d5c375e15\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") "
Aug 04 18:35:10.669515 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.669316    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7zbp\" (UniqueName: \"kubernetes.io/projected/3a67838d-68a3-4750-b04d-282d5c375e15-kube-api-access-z7zbp\") pod \"3a67838d-68a3-4750-b04d-282d5c375e15\" (UID: \"3a67838d-68a3-4750-b04d-282d5c375e15\") "
Aug 04 18:35:10.669647 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.669550    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a67838d-68a3-4750-b04d-282d5c375e15-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a67838d-68a3-4750-b04d-282d5c375e15" (UID: "3a67838d-68a3-4750-b04d-282d5c375e15"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:35:10.669647 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.669628    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a67838d-68a3-4750-b04d-282d5c375e15-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a67838d-68a3-4750-b04d-282d5c375e15" (UID: "3a67838d-68a3-4750-b04d-282d5c375e15"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:35:10.669737 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.669690    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a67838d-68a3-4750-b04d-282d5c375e15-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a67838d-68a3-4750-b04d-282d5c375e15" (UID: "3a67838d-68a3-4750-b04d-282d5c375e15"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:35:10.671279 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.671247    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a67838d-68a3-4750-b04d-282d5c375e15-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a67838d-68a3-4750-b04d-282d5c375e15" (UID: "3a67838d-68a3-4750-b04d-282d5c375e15"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:35:10.671406 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.671345    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a67838d-68a3-4750-b04d-282d5c375e15-kube-api-access-z7zbp" (OuterVolumeSpecName: "kube-api-access-z7zbp") pod "3a67838d-68a3-4750-b04d-282d5c375e15" (UID: "3a67838d-68a3-4750-b04d-282d5c375e15"). InnerVolumeSpecName "kube-api-access-z7zbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:35:10.770298 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.770215    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7zbp\" (UniqueName: \"kubernetes.io/projected/3a67838d-68a3-4750-b04d-282d5c375e15-kube-api-access-z7zbp\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:35:10.770298 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.770248    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a67838d-68a3-4750-b04d-282d5c375e15-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:35:10.770298 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.770262    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a67838d-68a3-4750-b04d-282d5c375e15-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:35:10.770298 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.770277    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a67838d-68a3-4750-b04d-282d5c375e15-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:35:10.770298 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:10.770290    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a67838d-68a3-4750-b04d-282d5c375e15-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:35:11.178905 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.178866    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"]
Aug 04 18:35:11.179262 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.179247    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="storage-initializer"
Aug 04 18:35:11.179312 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.179264    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="storage-initializer"
Aug 04 18:35:11.179312 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.179288    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kserve-container"
Aug 04 18:35:11.179312 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.179294    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kserve-container"
Aug 04 18:35:11.179312 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.179302    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kube-rbac-proxy"
Aug 04 18:35:11.179312 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.179309    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kube-rbac-proxy"
Aug 04 18:35:11.179498 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.179377    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kserve-container"
Aug 04 18:35:11.179498 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.179387    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" containerName="kube-rbac-proxy"
Aug 04 18:35:11.182833 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.182812    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.185846 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.185821    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Aug 04 18:35:11.191505 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.191487    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Aug 04 18:35:11.205283 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.205252    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"]
Aug 04 18:35:11.274987 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.274957    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87ff237e-92d3-4b89-ad97-9d5616aea180-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.274987 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.274988    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87ff237e-92d3-4b89-ad97-9d5616aea180-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.275193 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.275126    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7twl\" (UniqueName: \"kubernetes.io/projected/87ff237e-92d3-4b89-ad97-9d5616aea180-kube-api-access-n7twl\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.275193 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.275155    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87ff237e-92d3-4b89-ad97-9d5616aea180-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.275193 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.275181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87ff237e-92d3-4b89-ad97-9d5616aea180-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.375684 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.375650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7twl\" (UniqueName: \"kubernetes.io/projected/87ff237e-92d3-4b89-ad97-9d5616aea180-kube-api-access-n7twl\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.375684 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.375687    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87ff237e-92d3-4b89-ad97-9d5616aea180-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.375954 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.375720    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87ff237e-92d3-4b89-ad97-9d5616aea180-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.375954 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.375765    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87ff237e-92d3-4b89-ad97-9d5616aea180-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.375954 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.375789    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87ff237e-92d3-4b89-ad97-9d5616aea180-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.376162 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.376133    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87ff237e-92d3-4b89-ad97-9d5616aea180-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.376469 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.376449    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87ff237e-92d3-4b89-ad97-9d5616aea180-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.376545 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.376528    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87ff237e-92d3-4b89-ad97-9d5616aea180-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.378247 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.378227    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87ff237e-92d3-4b89-ad97-9d5616aea180-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.387579 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.387553    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7twl\" (UniqueName: \"kubernetes.io/projected/87ff237e-92d3-4b89-ad97-9d5616aea180-kube-api-access-n7twl\") pod \"isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.494290 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.494204    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:11.616605 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.616579    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"
Aug 04 18:35:11.641066 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.641038    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"]
Aug 04 18:35:11.642608 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:35:11.642586    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ff237e_92d3_4b89_ad97_9d5616aea180.slice/crio-d8a1f2e542f94e753edaf3ca2edf3ee838394ea4403280b904143ba5be0f1f4f WatchSource:0}: Error finding container d8a1f2e542f94e753edaf3ca2edf3ee838394ea4403280b904143ba5be0f1f4f: Status 404 returned error can't find the container with id d8a1f2e542f94e753edaf3ca2edf3ee838394ea4403280b904143ba5be0f1f4f
Aug 04 18:35:11.651721 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.651695    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"]
Aug 04 18:35:11.662884 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:11.662862    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-nm7ss"]
Aug 04 18:35:12.622897 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:12.622860    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" event={"ID":"87ff237e-92d3-4b89-ad97-9d5616aea180","Type":"ContainerStarted","Data":"3ed00dc632e4f392cac58973c8e3f433ff5b705fb7bfd65ad8ac71d87e7cfebc"}
Aug 04 18:35:12.623289 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:12.622934    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" event={"ID":"87ff237e-92d3-4b89-ad97-9d5616aea180","Type":"ContainerStarted","Data":"d8a1f2e542f94e753edaf3ca2edf3ee838394ea4403280b904143ba5be0f1f4f"}
Aug 04 18:35:13.194140 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:13.194108    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a67838d-68a3-4750-b04d-282d5c375e15" path="/var/lib/kubelet/pods/3a67838d-68a3-4750-b04d-282d5c375e15/volumes"
Aug 04 18:35:14.416456 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:14.416416    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Aug 04 18:35:15.638803 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:15.638724    2566 generic.go:356] "Generic (PLEG): container finished" podID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerID="3ed00dc632e4f392cac58973c8e3f433ff5b705fb7bfd65ad8ac71d87e7cfebc" exitCode=0
Aug 04 18:35:15.638803 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:15.638786    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" event={"ID":"87ff237e-92d3-4b89-ad97-9d5616aea180","Type":"ContainerDied","Data":"3ed00dc632e4f392cac58973c8e3f433ff5b705fb7bfd65ad8ac71d87e7cfebc"}
Aug 04 18:35:16.646417 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:16.646384    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" event={"ID":"87ff237e-92d3-4b89-ad97-9d5616aea180","Type":"ContainerStarted","Data":"16fd5d3668d458406f2b6f4054b730f380334bf3620922912ac075b2d40a5b96"}
Aug 04 18:35:16.646849 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:16.646446    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" event={"ID":"87ff237e-92d3-4b89-ad97-9d5616aea180","Type":"ContainerStarted","Data":"fb2ae95d3ebb7fe53b39ffddaf54a8150058256825eef6adab213965a1c7707e"}
Aug 04 18:35:16.646849 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:16.646730    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:16.646960 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:16.646896    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:16.648289 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:16.648262    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Aug 04 18:35:16.676081 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:16.676032    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" podStartSLOduration=2.077228615 podStartE2EDuration="5.676017578s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.598788963s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:35:16.674575866 +0000 UTC m=+2115.965190047" watchObservedRunningTime="2026-08-04 18:35:16.676017578 +0000 UTC m=+2115.966631749"
Aug 04 18:35:17.652393 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:17.652323    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Aug 04 18:35:22.657389 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:22.657346    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:35:22.657843 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:22.657817    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Aug 04 18:35:24.416587 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:24.416546    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Aug 04 18:35:32.658657 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:32.658616    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Aug 04 18:35:34.416944 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:34.416903    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Aug 04 18:35:42.658503 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:42.658466    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Aug 04 18:35:44.416523 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:44.416496    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:35:52.183047 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:52.183012    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"]
Aug 04 18:35:52.183476 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:52.183259    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kserve-container" containerID="cri-o://d1199441018912c99ea71a4874a38ecec9b716396434e760c8a23ad95672692a" gracePeriod=30
Aug 04 18:35:52.183476 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:52.183294    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kube-rbac-proxy" containerID="cri-o://b7695caf811398fe8d61a2c9ae0c533c9269b5d0355e9b474104f70a4ed7f809" gracePeriod=30
Aug 04 18:35:52.657906 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:52.657865    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Aug 04 18:35:52.822126 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:52.822094    2566 generic.go:356] "Generic (PLEG): container finished" podID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerID="b7695caf811398fe8d61a2c9ae0c533c9269b5d0355e9b474104f70a4ed7f809" exitCode=2
Aug 04 18:35:52.822298 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:52.822159    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" event={"ID":"155b1987-b73f-46f4-953f-6b8fdc2a0f51","Type":"ContainerDied","Data":"b7695caf811398fe8d61a2c9ae0c533c9269b5d0355e9b474104f70a4ed7f809"}
Aug 04 18:35:54.411132 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:54.411051    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Aug 04 18:35:54.416437 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:54.416411    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Aug 04 18:35:57.230001 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.229977    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:35:57.288562 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.288533    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/155b1987-b73f-46f4-953f-6b8fdc2a0f51-cabundle-cert\") pod \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") "
Aug 04 18:35:57.288760 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.288604    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/155b1987-b73f-46f4-953f-6b8fdc2a0f51-proxy-tls\") pod \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") "
Aug 04 18:35:57.288760 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.288631    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w69k8\" (UniqueName: \"kubernetes.io/projected/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kube-api-access-w69k8\") pod \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") "
Aug 04 18:35:57.288760 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.288666    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kserve-provision-location\") pod \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") "
Aug 04 18:35:57.288760 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.288681    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kube-rbac-proxy-sar-config\") pod \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\" (UID: \"155b1987-b73f-46f4-953f-6b8fdc2a0f51\") "
Aug 04 18:35:57.289221 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.289006    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155b1987-b73f-46f4-953f-6b8fdc2a0f51-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "155b1987-b73f-46f4-953f-6b8fdc2a0f51" (UID: "155b1987-b73f-46f4-953f-6b8fdc2a0f51"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:35:57.289221 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.289010    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "155b1987-b73f-46f4-953f-6b8fdc2a0f51" (UID: "155b1987-b73f-46f4-953f-6b8fdc2a0f51"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:35:57.289221 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.289175    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "155b1987-b73f-46f4-953f-6b8fdc2a0f51" (UID: "155b1987-b73f-46f4-953f-6b8fdc2a0f51"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:35:57.290868 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.290844    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/155b1987-b73f-46f4-953f-6b8fdc2a0f51-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "155b1987-b73f-46f4-953f-6b8fdc2a0f51" (UID: "155b1987-b73f-46f4-953f-6b8fdc2a0f51"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:35:57.290958 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.290892    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kube-api-access-w69k8" (OuterVolumeSpecName: "kube-api-access-w69k8") pod "155b1987-b73f-46f4-953f-6b8fdc2a0f51" (UID: "155b1987-b73f-46f4-953f-6b8fdc2a0f51"). InnerVolumeSpecName "kube-api-access-w69k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:35:57.390440 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.390404    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/155b1987-b73f-46f4-953f-6b8fdc2a0f51-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:35:57.390440 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.390433    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/155b1987-b73f-46f4-953f-6b8fdc2a0f51-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:35:57.390440 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.390442    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w69k8\" (UniqueName: \"kubernetes.io/projected/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kube-api-access-w69k8\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:35:57.390440 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.390452    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:35:57.390722 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.390461    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/155b1987-b73f-46f4-953f-6b8fdc2a0f51-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:35:57.528985 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.528946    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"]
Aug 04 18:35:57.529396 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.529379    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kube-rbac-proxy"
Aug 04 18:35:57.529500 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.529399    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kube-rbac-proxy"
Aug 04 18:35:57.529500 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.529421    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="storage-initializer"
Aug 04 18:35:57.529500 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.529429    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="storage-initializer"
Aug 04 18:35:57.529500 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.529458    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kserve-container"
Aug 04 18:35:57.529500 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.529467    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kserve-container"
Aug 04 18:35:57.529764 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.529563    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kserve-container"
Aug 04 18:35:57.529764 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.529579    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerName="kube-rbac-proxy"
Aug 04 18:35:57.533088 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.533068    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.537759 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.537740    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Aug 04 18:35:57.538717 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.538699    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Aug 04 18:35:57.555934 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.555900    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"]
Aug 04 18:35:57.591889 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.591862    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baa178b4-d81f-40c9-a025-0e5f8010697e-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-n87rs\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.592065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.591900    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baa178b4-d81f-40c9-a025-0e5f8010697e-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-n87rs\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.592065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.591921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baa178b4-d81f-40c9-a025-0e5f8010697e-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-n87rs\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.592065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.591941    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baa178b4-d81f-40c9-a025-0e5f8010697e-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-n87rs\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.592065 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.592038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ggxm\" (UniqueName: \"kubernetes.io/projected/baa178b4-d81f-40c9-a025-0e5f8010697e-kube-api-access-9ggxm\") pod \"isvc-tensorflow-predictor-547dfc5669-n87rs\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.692531 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.692440    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ggxm\" (UniqueName: \"kubernetes.io/projected/baa178b4-d81f-40c9-a025-0e5f8010697e-kube-api-access-9ggxm\") pod \"isvc-tensorflow-predictor-547dfc5669-n87rs\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.692531 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.692531    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baa178b4-d81f-40c9-a025-0e5f8010697e-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-n87rs\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.692764 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.692559    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baa178b4-d81f-40c9-a025-0e5f8010697e-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-n87rs\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.692764 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.692579    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baa178b4-d81f-40c9-a025-0e5f8010697e-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-n87rs\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.692764 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.692597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baa178b4-d81f-40c9-a025-0e5f8010697e-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-n87rs\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.692967 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.692950    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baa178b4-d81f-40c9-a025-0e5f8010697e-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-n87rs\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.693402 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.693382    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baa178b4-d81f-40c9-a025-0e5f8010697e-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-n87rs\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.693456 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.693382    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baa178b4-d81f-40c9-a025-0e5f8010697e-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-n87rs\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.695099 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.695078    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baa178b4-d81f-40c9-a025-0e5f8010697e-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-n87rs\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.707012 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.706987    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ggxm\" (UniqueName: \"kubernetes.io/projected/baa178b4-d81f-40c9-a025-0e5f8010697e-kube-api-access-9ggxm\") pod \"isvc-tensorflow-predictor-547dfc5669-n87rs\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.843096 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.843061    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:35:57.847253 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.847229    2566 generic.go:356] "Generic (PLEG): container finished" podID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" containerID="d1199441018912c99ea71a4874a38ecec9b716396434e760c8a23ad95672692a" exitCode=0
Aug 04 18:35:57.847411 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.847308    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"
Aug 04 18:35:57.847411 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.847301    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" event={"ID":"155b1987-b73f-46f4-953f-6b8fdc2a0f51","Type":"ContainerDied","Data":"d1199441018912c99ea71a4874a38ecec9b716396434e760c8a23ad95672692a"}
Aug 04 18:35:57.847526 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.847431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62" event={"ID":"155b1987-b73f-46f4-953f-6b8fdc2a0f51","Type":"ContainerDied","Data":"b7bd346cecce48fbd4d5b088bbf8e205b40a5cc9b946315f380e1fb702005338"}
Aug 04 18:35:57.847526 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.847460    2566 scope.go:117] "RemoveContainer" containerID="b7695caf811398fe8d61a2c9ae0c533c9269b5d0355e9b474104f70a4ed7f809"
Aug 04 18:35:57.856221 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.856195    2566 scope.go:117] "RemoveContainer" containerID="d1199441018912c99ea71a4874a38ecec9b716396434e760c8a23ad95672692a"
Aug 04 18:35:57.864130 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.864105    2566 scope.go:117] "RemoveContainer" containerID="9fc4a891f6d16bd6d8a888bdb4ba26bab58a733ba677e9171342d18d5033e4ae"
Aug 04 18:35:57.872174 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.872155    2566 scope.go:117] "RemoveContainer" containerID="b7695caf811398fe8d61a2c9ae0c533c9269b5d0355e9b474104f70a4ed7f809"
Aug 04 18:35:57.872517 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:35:57.872499    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7695caf811398fe8d61a2c9ae0c533c9269b5d0355e9b474104f70a4ed7f809\": container with ID starting with b7695caf811398fe8d61a2c9ae0c533c9269b5d0355e9b474104f70a4ed7f809 not found: ID does not exist" containerID="b7695caf811398fe8d61a2c9ae0c533c9269b5d0355e9b474104f70a4ed7f809"
Aug 04 18:35:57.872609 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.872525    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7695caf811398fe8d61a2c9ae0c533c9269b5d0355e9b474104f70a4ed7f809"} err="failed to get container status \"b7695caf811398fe8d61a2c9ae0c533c9269b5d0355e9b474104f70a4ed7f809\": rpc error: code = NotFound desc = could not find container \"b7695caf811398fe8d61a2c9ae0c533c9269b5d0355e9b474104f70a4ed7f809\": container with ID starting with b7695caf811398fe8d61a2c9ae0c533c9269b5d0355e9b474104f70a4ed7f809 not found: ID does not exist"
Aug 04 18:35:57.872609 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.872541    2566 scope.go:117] "RemoveContainer" containerID="d1199441018912c99ea71a4874a38ecec9b716396434e760c8a23ad95672692a"
Aug 04 18:35:57.872822 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:35:57.872799    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1199441018912c99ea71a4874a38ecec9b716396434e760c8a23ad95672692a\": container with ID starting with d1199441018912c99ea71a4874a38ecec9b716396434e760c8a23ad95672692a not found: ID does not exist" containerID="d1199441018912c99ea71a4874a38ecec9b716396434e760c8a23ad95672692a"
Aug 04 18:35:57.872865 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.872832    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1199441018912c99ea71a4874a38ecec9b716396434e760c8a23ad95672692a"} err="failed to get container status \"d1199441018912c99ea71a4874a38ecec9b716396434e760c8a23ad95672692a\": rpc error: code = NotFound desc = could not find container \"d1199441018912c99ea71a4874a38ecec9b716396434e760c8a23ad95672692a\": container with ID starting with d1199441018912c99ea71a4874a38ecec9b716396434e760c8a23ad95672692a not found: ID does not exist"
Aug 04 18:35:57.872865 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.872855    2566 scope.go:117] "RemoveContainer" containerID="9fc4a891f6d16bd6d8a888bdb4ba26bab58a733ba677e9171342d18d5033e4ae"
Aug 04 18:35:57.873114 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:35:57.873099    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc4a891f6d16bd6d8a888bdb4ba26bab58a733ba677e9171342d18d5033e4ae\": container with ID starting with 9fc4a891f6d16bd6d8a888bdb4ba26bab58a733ba677e9171342d18d5033e4ae not found: ID does not exist" containerID="9fc4a891f6d16bd6d8a888bdb4ba26bab58a733ba677e9171342d18d5033e4ae"
Aug 04 18:35:57.873177 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.873120    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc4a891f6d16bd6d8a888bdb4ba26bab58a733ba677e9171342d18d5033e4ae"} err="failed to get container status \"9fc4a891f6d16bd6d8a888bdb4ba26bab58a733ba677e9171342d18d5033e4ae\": rpc error: code = NotFound desc = could not find container \"9fc4a891f6d16bd6d8a888bdb4ba26bab58a733ba677e9171342d18d5033e4ae\": container with ID starting with 9fc4a891f6d16bd6d8a888bdb4ba26bab58a733ba677e9171342d18d5033e4ae not found: ID does not exist"
Aug 04 18:35:57.891236 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.891208    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"]
Aug 04 18:35:57.899655 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:57.899625    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5d8958df4-gvp62"]
Aug 04 18:35:58.002071 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:58.002048    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"]
Aug 04 18:35:58.003537 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:35:58.003515    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa178b4_d81f_40c9_a025_0e5f8010697e.slice/crio-2debbfe16d13eaf577d9fa2c6269727792f6a3f1b18a6a76ff45d1c7c66b4160 WatchSource:0}: Error finding container 2debbfe16d13eaf577d9fa2c6269727792f6a3f1b18a6a76ff45d1c7c66b4160: Status 404 returned error can't find the container with id 2debbfe16d13eaf577d9fa2c6269727792f6a3f1b18a6a76ff45d1c7c66b4160
Aug 04 18:35:58.855527 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:58.855493    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" event={"ID":"baa178b4-d81f-40c9-a025-0e5f8010697e","Type":"ContainerStarted","Data":"ac0246c89e147d8c748816d21dfa87794f2fc2cefa0d7500ce538f38ea2f6915"}
Aug 04 18:35:58.855916 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:58.855561    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" event={"ID":"baa178b4-d81f-40c9-a025-0e5f8010697e","Type":"ContainerStarted","Data":"2debbfe16d13eaf577d9fa2c6269727792f6a3f1b18a6a76ff45d1c7c66b4160"}
Aug 04 18:35:59.194520 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:35:59.194445    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="155b1987-b73f-46f4-953f-6b8fdc2a0f51" path="/var/lib/kubelet/pods/155b1987-b73f-46f4-953f-6b8fdc2a0f51/volumes"
Aug 04 18:36:02.658665 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:02.658570    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Aug 04 18:36:02.876346 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:02.876296    2566 generic.go:356] "Generic (PLEG): container finished" podID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerID="ac0246c89e147d8c748816d21dfa87794f2fc2cefa0d7500ce538f38ea2f6915" exitCode=0
Aug 04 18:36:02.876525 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:02.876380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" event={"ID":"baa178b4-d81f-40c9-a025-0e5f8010697e","Type":"ContainerDied","Data":"ac0246c89e147d8c748816d21dfa87794f2fc2cefa0d7500ce538f38ea2f6915"}
Aug 04 18:36:02.877638 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:02.877624    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:36:06.901413 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:06.901384    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" event={"ID":"baa178b4-d81f-40c9-a025-0e5f8010697e","Type":"ContainerStarted","Data":"ad92461ee462698d19ec96708c946404661ec314a4faf2c0eb15559273df8b6c"}
Aug 04 18:36:07.908659 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:07.908620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" event={"ID":"baa178b4-d81f-40c9-a025-0e5f8010697e","Type":"ContainerStarted","Data":"a3dcab57d642d655e1c413868c1c2934baf69ed809aa6651fe72d1f5450088c7"}
Aug 04 18:36:07.909044 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:07.908695    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:36:07.933037 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:07.932984    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" podStartSLOduration=2.710816907 podStartE2EDuration="10.932969986s" totalImagesPullingTime="3.913296978s" totalInitContainerRuntime="4.308856101s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:36:07.93147523 +0000 UTC m=+2167.222089418" watchObservedRunningTime="2026-08-04 18:36:07.932969986 +0000 UTC m=+2167.223584156"
Aug 04 18:36:08.913972 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:08.913945    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:36:08.914931 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:08.914906    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Aug 04 18:36:09.918554 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:09.918508    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Aug 04 18:36:12.658507 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:12.658471    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Aug 04 18:36:14.923177 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:14.923147    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:36:14.923694 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:14.923667    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Aug 04 18:36:22.657826 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:22.657777    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Aug 04 18:36:24.924722 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:24.924696    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:36:32.658506 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:32.658473    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:36:38.606320 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:38.606284    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"]
Aug 04 18:36:38.606871 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:38.606554    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kserve-container" containerID="cri-o://ad92461ee462698d19ec96708c946404661ec314a4faf2c0eb15559273df8b6c" gracePeriod=30
Aug 04 18:36:38.606871 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:38.606582    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kube-rbac-proxy" containerID="cri-o://a3dcab57d642d655e1c413868c1c2934baf69ed809aa6651fe72d1f5450088c7" gracePeriod=30
Aug 04 18:36:39.055726 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:39.055693    2566 generic.go:356] "Generic (PLEG): container finished" podID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerID="a3dcab57d642d655e1c413868c1c2934baf69ed809aa6651fe72d1f5450088c7" exitCode=2
Aug 04 18:36:39.055885 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:39.055763    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" event={"ID":"baa178b4-d81f-40c9-a025-0e5f8010697e","Type":"ContainerDied","Data":"a3dcab57d642d655e1c413868c1c2934baf69ed809aa6651fe72d1f5450088c7"}
Aug 04 18:36:39.919309 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:39.919265    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Aug 04 18:36:40.973783 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:40.973751    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"]
Aug 04 18:36:40.974233 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:40.973970    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kserve-container" containerID="cri-o://fb2ae95d3ebb7fe53b39ffddaf54a8150058256825eef6adab213965a1c7707e" gracePeriod=30
Aug 04 18:36:40.974233 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:40.974011    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kube-rbac-proxy" containerID="cri-o://16fd5d3668d458406f2b6f4054b730f380334bf3620922912ac075b2d40a5b96" gracePeriod=30
Aug 04 18:36:41.044734 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.044694    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"]
Aug 04 18:36:41.047575 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.047554    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.050293 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.050274    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Aug 04 18:36:41.061932 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.061902    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"]
Aug 04 18:36:41.106597 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.106558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-gf8fp\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.106597 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.106596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-gf8fp\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.106775 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.106696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0808b6a-104b-4486-a7bd-3eef07f71b7d-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-gf8fp\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.106775 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.106722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0808b6a-104b-4486-a7bd-3eef07f71b7d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-gf8fp\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.106867 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.106804    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5f59\" (UniqueName: \"kubernetes.io/projected/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kube-api-access-w5f59\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-gf8fp\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.207195 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.207164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0808b6a-104b-4486-a7bd-3eef07f71b7d-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-gf8fp\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.207195 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.207198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0808b6a-104b-4486-a7bd-3eef07f71b7d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-gf8fp\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.207452 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.207238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5f59\" (UniqueName: \"kubernetes.io/projected/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kube-api-access-w5f59\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-gf8fp\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.207452 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.207272    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-gf8fp\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.207452 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.207290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-gf8fp\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.207838 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.207807    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-gf8fp\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.208024 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.208005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-gf8fp\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.208088 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.208011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0808b6a-104b-4486-a7bd-3eef07f71b7d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-gf8fp\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.209656 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.209634    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0808b6a-104b-4486-a7bd-3eef07f71b7d-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-gf8fp\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.217431 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.217411    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5f59\" (UniqueName: \"kubernetes.io/projected/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kube-api-access-w5f59\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-gf8fp\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.364551 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.364516    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:41.504211 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:41.504178    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"]
Aug 04 18:36:41.505168 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:36:41.505144    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0808b6a_104b_4486_a7bd_3eef07f71b7d.slice/crio-51e1c8981f8370078655ec0e35c878c08777ac790b439d699a24506fa9211d48 WatchSource:0}: Error finding container 51e1c8981f8370078655ec0e35c878c08777ac790b439d699a24506fa9211d48: Status 404 returned error can't find the container with id 51e1c8981f8370078655ec0e35c878c08777ac790b439d699a24506fa9211d48
Aug 04 18:36:42.072909 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:42.072877    2566 generic.go:356] "Generic (PLEG): container finished" podID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerID="16fd5d3668d458406f2b6f4054b730f380334bf3620922912ac075b2d40a5b96" exitCode=2
Aug 04 18:36:42.073406 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:42.072951    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" event={"ID":"87ff237e-92d3-4b89-ad97-9d5616aea180","Type":"ContainerDied","Data":"16fd5d3668d458406f2b6f4054b730f380334bf3620922912ac075b2d40a5b96"}
Aug 04 18:36:42.074291 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:42.074268    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" event={"ID":"d0808b6a-104b-4486-a7bd-3eef07f71b7d","Type":"ContainerStarted","Data":"09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c"}
Aug 04 18:36:42.074443 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:42.074320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" event={"ID":"d0808b6a-104b-4486-a7bd-3eef07f71b7d","Type":"ContainerStarted","Data":"51e1c8981f8370078655ec0e35c878c08777ac790b439d699a24506fa9211d48"}
Aug 04 18:36:42.653268 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:42.653228    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Aug 04 18:36:42.658567 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:42.658545    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Aug 04 18:36:44.919327 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:44.919276    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Aug 04 18:36:45.917140 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:45.917115    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:36:46.049532 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.049438    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87ff237e-92d3-4b89-ad97-9d5616aea180-proxy-tls\") pod \"87ff237e-92d3-4b89-ad97-9d5616aea180\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") "
Aug 04 18:36:46.049532 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.049495    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7twl\" (UniqueName: \"kubernetes.io/projected/87ff237e-92d3-4b89-ad97-9d5616aea180-kube-api-access-n7twl\") pod \"87ff237e-92d3-4b89-ad97-9d5616aea180\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") "
Aug 04 18:36:46.050042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.049568    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87ff237e-92d3-4b89-ad97-9d5616aea180-kube-rbac-proxy-sar-config\") pod \"87ff237e-92d3-4b89-ad97-9d5616aea180\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") "
Aug 04 18:36:46.050042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.049627    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87ff237e-92d3-4b89-ad97-9d5616aea180-kserve-provision-location\") pod \"87ff237e-92d3-4b89-ad97-9d5616aea180\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") "
Aug 04 18:36:46.050042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.049696    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87ff237e-92d3-4b89-ad97-9d5616aea180-cabundle-cert\") pod \"87ff237e-92d3-4b89-ad97-9d5616aea180\" (UID: \"87ff237e-92d3-4b89-ad97-9d5616aea180\") "
Aug 04 18:36:46.050042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.049967    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ff237e-92d3-4b89-ad97-9d5616aea180-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87ff237e-92d3-4b89-ad97-9d5616aea180" (UID: "87ff237e-92d3-4b89-ad97-9d5616aea180"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:36:46.050042 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.049989    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ff237e-92d3-4b89-ad97-9d5616aea180-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "87ff237e-92d3-4b89-ad97-9d5616aea180" (UID: "87ff237e-92d3-4b89-ad97-9d5616aea180"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:36:46.050229 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.050101    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ff237e-92d3-4b89-ad97-9d5616aea180-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "87ff237e-92d3-4b89-ad97-9d5616aea180" (UID: "87ff237e-92d3-4b89-ad97-9d5616aea180"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:36:46.052087 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.052062    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ff237e-92d3-4b89-ad97-9d5616aea180-kube-api-access-n7twl" (OuterVolumeSpecName: "kube-api-access-n7twl") pod "87ff237e-92d3-4b89-ad97-9d5616aea180" (UID: "87ff237e-92d3-4b89-ad97-9d5616aea180"). InnerVolumeSpecName "kube-api-access-n7twl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:36:46.052087 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.052076    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ff237e-92d3-4b89-ad97-9d5616aea180-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "87ff237e-92d3-4b89-ad97-9d5616aea180" (UID: "87ff237e-92d3-4b89-ad97-9d5616aea180"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:36:46.094032 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.093999    2566 generic.go:356] "Generic (PLEG): container finished" podID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerID="fb2ae95d3ebb7fe53b39ffddaf54a8150058256825eef6adab213965a1c7707e" exitCode=0
Aug 04 18:36:46.094172 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.094067    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"
Aug 04 18:36:46.094172 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.094078    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" event={"ID":"87ff237e-92d3-4b89-ad97-9d5616aea180","Type":"ContainerDied","Data":"fb2ae95d3ebb7fe53b39ffddaf54a8150058256825eef6adab213965a1c7707e"}
Aug 04 18:36:46.094172 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.094134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w" event={"ID":"87ff237e-92d3-4b89-ad97-9d5616aea180","Type":"ContainerDied","Data":"d8a1f2e542f94e753edaf3ca2edf3ee838394ea4403280b904143ba5be0f1f4f"}
Aug 04 18:36:46.094172 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.094163    2566 scope.go:117] "RemoveContainer" containerID="16fd5d3668d458406f2b6f4054b730f380334bf3620922912ac075b2d40a5b96"
Aug 04 18:36:46.095837 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.095810    2566 generic.go:356] "Generic (PLEG): container finished" podID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerID="09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c" exitCode=0
Aug 04 18:36:46.095942 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.095880    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" event={"ID":"d0808b6a-104b-4486-a7bd-3eef07f71b7d","Type":"ContainerDied","Data":"09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c"}
Aug 04 18:36:46.103427 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.103407    2566 scope.go:117] "RemoveContainer" containerID="fb2ae95d3ebb7fe53b39ffddaf54a8150058256825eef6adab213965a1c7707e"
Aug 04 18:36:46.111737 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.111716    2566 scope.go:117] "RemoveContainer" containerID="3ed00dc632e4f392cac58973c8e3f433ff5b705fb7bfd65ad8ac71d87e7cfebc"
Aug 04 18:36:46.121540 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.121522    2566 scope.go:117] "RemoveContainer" containerID="16fd5d3668d458406f2b6f4054b730f380334bf3620922912ac075b2d40a5b96"
Aug 04 18:36:46.121836 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:36:46.121817    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fd5d3668d458406f2b6f4054b730f380334bf3620922912ac075b2d40a5b96\": container with ID starting with 16fd5d3668d458406f2b6f4054b730f380334bf3620922912ac075b2d40a5b96 not found: ID does not exist" containerID="16fd5d3668d458406f2b6f4054b730f380334bf3620922912ac075b2d40a5b96"
Aug 04 18:36:46.121916 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.121849    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fd5d3668d458406f2b6f4054b730f380334bf3620922912ac075b2d40a5b96"} err="failed to get container status \"16fd5d3668d458406f2b6f4054b730f380334bf3620922912ac075b2d40a5b96\": rpc error: code = NotFound desc = could not find container \"16fd5d3668d458406f2b6f4054b730f380334bf3620922912ac075b2d40a5b96\": container with ID starting with 16fd5d3668d458406f2b6f4054b730f380334bf3620922912ac075b2d40a5b96 not found: ID does not exist"
Aug 04 18:36:46.121916 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.121872    2566 scope.go:117] "RemoveContainer" containerID="fb2ae95d3ebb7fe53b39ffddaf54a8150058256825eef6adab213965a1c7707e"
Aug 04 18:36:46.122105 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:36:46.122088    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb2ae95d3ebb7fe53b39ffddaf54a8150058256825eef6adab213965a1c7707e\": container with ID starting with fb2ae95d3ebb7fe53b39ffddaf54a8150058256825eef6adab213965a1c7707e not found: ID does not exist" containerID="fb2ae95d3ebb7fe53b39ffddaf54a8150058256825eef6adab213965a1c7707e"
Aug 04 18:36:46.122145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.122111    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2ae95d3ebb7fe53b39ffddaf54a8150058256825eef6adab213965a1c7707e"} err="failed to get container status \"fb2ae95d3ebb7fe53b39ffddaf54a8150058256825eef6adab213965a1c7707e\": rpc error: code = NotFound desc = could not find container \"fb2ae95d3ebb7fe53b39ffddaf54a8150058256825eef6adab213965a1c7707e\": container with ID starting with fb2ae95d3ebb7fe53b39ffddaf54a8150058256825eef6adab213965a1c7707e not found: ID does not exist"
Aug 04 18:36:46.122145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.122127    2566 scope.go:117] "RemoveContainer" containerID="3ed00dc632e4f392cac58973c8e3f433ff5b705fb7bfd65ad8ac71d87e7cfebc"
Aug 04 18:36:46.122405 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:36:46.122387    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ed00dc632e4f392cac58973c8e3f433ff5b705fb7bfd65ad8ac71d87e7cfebc\": container with ID starting with 3ed00dc632e4f392cac58973c8e3f433ff5b705fb7bfd65ad8ac71d87e7cfebc not found: ID does not exist" containerID="3ed00dc632e4f392cac58973c8e3f433ff5b705fb7bfd65ad8ac71d87e7cfebc"
Aug 04 18:36:46.122467 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.122411    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ed00dc632e4f392cac58973c8e3f433ff5b705fb7bfd65ad8ac71d87e7cfebc"} err="failed to get container status \"3ed00dc632e4f392cac58973c8e3f433ff5b705fb7bfd65ad8ac71d87e7cfebc\": rpc error: code = NotFound desc = could not find container \"3ed00dc632e4f392cac58973c8e3f433ff5b705fb7bfd65ad8ac71d87e7cfebc\": container with ID starting with 3ed00dc632e4f392cac58973c8e3f433ff5b705fb7bfd65ad8ac71d87e7cfebc not found: ID does not exist"
Aug 04 18:36:46.140667 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.140636    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"]
Aug 04 18:36:46.142254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.142234    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5ff4dcd9c9-xcv9w"]
Aug 04 18:36:46.151335 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.151315    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87ff237e-92d3-4b89-ad97-9d5616aea180-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:36:46.151335 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.151336    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87ff237e-92d3-4b89-ad97-9d5616aea180-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:36:46.151492 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.151345    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87ff237e-92d3-4b89-ad97-9d5616aea180-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:36:46.151492 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.151377    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87ff237e-92d3-4b89-ad97-9d5616aea180-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:36:46.151492 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:46.151385    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7twl\" (UniqueName: \"kubernetes.io/projected/87ff237e-92d3-4b89-ad97-9d5616aea180-kube-api-access-n7twl\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:36:47.103593 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:47.103559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" event={"ID":"d0808b6a-104b-4486-a7bd-3eef07f71b7d","Type":"ContainerStarted","Data":"fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399"}
Aug 04 18:36:47.104047 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:47.103663    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" event={"ID":"d0808b6a-104b-4486-a7bd-3eef07f71b7d","Type":"ContainerStarted","Data":"f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545"}
Aug 04 18:36:47.104047 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:47.103938    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:47.104161 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:47.104089    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:47.105521 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:47.105492    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Aug 04 18:36:47.167454 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:47.167400    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podStartSLOduration=2.360212473 podStartE2EDuration="6.167383596s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.807171123s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:36:47.165113659 +0000 UTC m=+2206.455727829" watchObservedRunningTime="2026-08-04 18:36:47.167383596 +0000 UTC m=+2206.457997766"
Aug 04 18:36:47.196172 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:47.196142    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" path="/var/lib/kubelet/pods/87ff237e-92d3-4b89-ad97-9d5616aea180/volumes"
Aug 04 18:36:48.110774 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:48.110731    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Aug 04 18:36:49.918822 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:49.918775    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Aug 04 18:36:49.919310 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:49.918893    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:36:53.115739 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:53.115710    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:36:53.116229 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:53.116202    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Aug 04 18:36:54.918998 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:54.918960    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Aug 04 18:36:59.918957 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:36:59.918904    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Aug 04 18:37:03.116170 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:03.116125    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Aug 04 18:37:04.919621 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:04.919575    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Aug 04 18:37:09.217888 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.216040    2566 generic.go:356] "Generic (PLEG): container finished" podID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerID="ad92461ee462698d19ec96708c946404661ec314a4faf2c0eb15559273df8b6c" exitCode=137
Aug 04 18:37:09.217888 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.216089    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" event={"ID":"baa178b4-d81f-40c9-a025-0e5f8010697e","Type":"ContainerDied","Data":"ad92461ee462698d19ec96708c946404661ec314a4faf2c0eb15559273df8b6c"}
Aug 04 18:37:09.274824 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.274481    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:37:09.347791 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.347147    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ggxm\" (UniqueName: \"kubernetes.io/projected/baa178b4-d81f-40c9-a025-0e5f8010697e-kube-api-access-9ggxm\") pod \"baa178b4-d81f-40c9-a025-0e5f8010697e\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") "
Aug 04 18:37:09.347791 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.347202    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baa178b4-d81f-40c9-a025-0e5f8010697e-cabundle-cert\") pod \"baa178b4-d81f-40c9-a025-0e5f8010697e\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") "
Aug 04 18:37:09.347791 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.347241    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baa178b4-d81f-40c9-a025-0e5f8010697e-kserve-provision-location\") pod \"baa178b4-d81f-40c9-a025-0e5f8010697e\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") "
Aug 04 18:37:09.347791 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.347309    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baa178b4-d81f-40c9-a025-0e5f8010697e-kube-rbac-proxy-sar-config\") pod \"baa178b4-d81f-40c9-a025-0e5f8010697e\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") "
Aug 04 18:37:09.347791 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.347374    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baa178b4-d81f-40c9-a025-0e5f8010697e-proxy-tls\") pod \"baa178b4-d81f-40c9-a025-0e5f8010697e\" (UID: \"baa178b4-d81f-40c9-a025-0e5f8010697e\") "
Aug 04 18:37:09.348211 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.348081    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa178b4-d81f-40c9-a025-0e5f8010697e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "baa178b4-d81f-40c9-a025-0e5f8010697e" (UID: "baa178b4-d81f-40c9-a025-0e5f8010697e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:37:09.348211 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.348173    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa178b4-d81f-40c9-a025-0e5f8010697e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "baa178b4-d81f-40c9-a025-0e5f8010697e" (UID: "baa178b4-d81f-40c9-a025-0e5f8010697e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:37:09.349885 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.349849    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baa178b4-d81f-40c9-a025-0e5f8010697e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "baa178b4-d81f-40c9-a025-0e5f8010697e" (UID: "baa178b4-d81f-40c9-a025-0e5f8010697e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:37:09.350605 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.350585    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa178b4-d81f-40c9-a025-0e5f8010697e-kube-api-access-9ggxm" (OuterVolumeSpecName: "kube-api-access-9ggxm") pod "baa178b4-d81f-40c9-a025-0e5f8010697e" (UID: "baa178b4-d81f-40c9-a025-0e5f8010697e"). InnerVolumeSpecName "kube-api-access-9ggxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:37:09.361850 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.361800    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa178b4-d81f-40c9-a025-0e5f8010697e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "baa178b4-d81f-40c9-a025-0e5f8010697e" (UID: "baa178b4-d81f-40c9-a025-0e5f8010697e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:37:09.448119 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.448029    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baa178b4-d81f-40c9-a025-0e5f8010697e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:37:09.448119 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.448058    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baa178b4-d81f-40c9-a025-0e5f8010697e-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:37:09.448119 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.448068    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ggxm\" (UniqueName: \"kubernetes.io/projected/baa178b4-d81f-40c9-a025-0e5f8010697e-kube-api-access-9ggxm\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:37:09.448119 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.448077    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baa178b4-d81f-40c9-a025-0e5f8010697e-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:37:09.448119 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:09.448087    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baa178b4-d81f-40c9-a025-0e5f8010697e-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:37:10.222804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:10.222777    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"
Aug 04 18:37:10.223266 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:10.222779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs" event={"ID":"baa178b4-d81f-40c9-a025-0e5f8010697e","Type":"ContainerDied","Data":"2debbfe16d13eaf577d9fa2c6269727792f6a3f1b18a6a76ff45d1c7c66b4160"}
Aug 04 18:37:10.223266 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:10.222916    2566 scope.go:117] "RemoveContainer" containerID="a3dcab57d642d655e1c413868c1c2934baf69ed809aa6651fe72d1f5450088c7"
Aug 04 18:37:10.231671 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:10.231654    2566 scope.go:117] "RemoveContainer" containerID="ad92461ee462698d19ec96708c946404661ec314a4faf2c0eb15559273df8b6c"
Aug 04 18:37:10.239218 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:10.239199    2566 scope.go:117] "RemoveContainer" containerID="ac0246c89e147d8c748816d21dfa87794f2fc2cefa0d7500ce538f38ea2f6915"
Aug 04 18:37:10.245698 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:10.245675    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"]
Aug 04 18:37:10.252837 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:10.252810    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-n87rs"]
Aug 04 18:37:11.202150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:11.202115    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" path="/var/lib/kubelet/pods/baa178b4-d81f-40c9-a025-0e5f8010697e/volumes"
Aug 04 18:37:13.116208 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:13.116164    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Aug 04 18:37:23.116266 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:23.116183    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Aug 04 18:37:30.723845 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.723813    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"]
Aug 04 18:37:30.724287 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.724271    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kube-rbac-proxy"
Aug 04 18:37:30.724382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.724289    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kube-rbac-proxy"
Aug 04 18:37:30.724382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.724303    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kube-rbac-proxy"
Aug 04 18:37:30.724382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.724309    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kube-rbac-proxy"
Aug 04 18:37:30.724382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.724321    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="storage-initializer"
Aug 04 18:37:30.724382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.724327    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="storage-initializer"
Aug 04 18:37:30.724382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.724334    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="storage-initializer"
Aug 04 18:37:30.724382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.724339    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="storage-initializer"
Aug 04 18:37:30.724382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.724348    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kserve-container"
Aug 04 18:37:30.724382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.724375    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kserve-container"
Aug 04 18:37:30.724382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.724382    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kserve-container"
Aug 04 18:37:30.724382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.724387    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kserve-container"
Aug 04 18:37:30.724771 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.724454    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kube-rbac-proxy"
Aug 04 18:37:30.724771 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.724466    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kserve-container"
Aug 04 18:37:30.724771 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.724472    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="baa178b4-d81f-40c9-a025-0e5f8010697e" containerName="kserve-container"
Aug 04 18:37:30.724771 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.724479    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ff237e-92d3-4b89-ad97-9d5616aea180" containerName="kube-rbac-proxy"
Aug 04 18:37:30.729438 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.729419    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:30.731791 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.731766    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Aug 04 18:37:30.731943 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.731776    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Aug 04 18:37:30.738840 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.738819    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"]
Aug 04 18:37:30.845194 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.845159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6v9t\" (UniqueName: \"kubernetes.io/projected/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kube-api-access-b6v9t\") pod \"isvc-triton-predictor-5d7db58676-59slr\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:30.845410 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.845210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-59slr\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:30.845410 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.845230    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d42fc83c-2ce5-483b-826c-d66ac55d65d7-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-59slr\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:30.845410 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.845329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d42fc83c-2ce5-483b-826c-d66ac55d65d7-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-59slr\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:30.845410 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.845343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-59slr\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:30.946244 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.946208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-59slr\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:30.946244 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.946246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d42fc83c-2ce5-483b-826c-d66ac55d65d7-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-59slr\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:30.946548 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.946475    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d42fc83c-2ce5-483b-826c-d66ac55d65d7-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-59slr\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:30.946548 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.946514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-59slr\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:30.946667 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.946555    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6v9t\" (UniqueName: \"kubernetes.io/projected/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kube-api-access-b6v9t\") pod \"isvc-triton-predictor-5d7db58676-59slr\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:30.946959 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.946926    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-59slr\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:30.947064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.946937    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-59slr\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:30.947064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.947052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d42fc83c-2ce5-483b-826c-d66ac55d65d7-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-59slr\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:30.949009 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.948988    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d42fc83c-2ce5-483b-826c-d66ac55d65d7-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-59slr\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:30.955663 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:30.955643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6v9t\" (UniqueName: \"kubernetes.io/projected/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kube-api-access-b6v9t\") pod \"isvc-triton-predictor-5d7db58676-59slr\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:31.040294 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:31.040206    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:37:31.180014 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:31.179989    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"]
Aug 04 18:37:31.182211 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:37:31.182174    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42fc83c_2ce5_483b_826c_d66ac55d65d7.slice/crio-b3ed036de45ecbf0646d242f25081e80166ab2763ca7cf47fdc7c26c6b82b4d4 WatchSource:0}: Error finding container b3ed036de45ecbf0646d242f25081e80166ab2763ca7cf47fdc7c26c6b82b4d4: Status 404 returned error can't find the container with id b3ed036de45ecbf0646d242f25081e80166ab2763ca7cf47fdc7c26c6b82b4d4
Aug 04 18:37:31.323002 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:31.322965    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr" event={"ID":"d42fc83c-2ce5-483b-826c-d66ac55d65d7","Type":"ContainerStarted","Data":"daf4579bd1d61d6622ab1bb716ec5e7b397e96cbe75b5a02fad3f5a891946c87"}
Aug 04 18:37:31.323146 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:31.323038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr" event={"ID":"d42fc83c-2ce5-483b-826c-d66ac55d65d7","Type":"ContainerStarted","Data":"b3ed036de45ecbf0646d242f25081e80166ab2763ca7cf47fdc7c26c6b82b4d4"}
Aug 04 18:37:33.117144 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:33.117105    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Aug 04 18:37:35.343967 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:35.343937    2566 generic.go:356] "Generic (PLEG): container finished" podID="d42fc83c-2ce5-483b-826c-d66ac55d65d7" containerID="daf4579bd1d61d6622ab1bb716ec5e7b397e96cbe75b5a02fad3f5a891946c87" exitCode=0
Aug 04 18:37:35.344332 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:35.344017    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr" event={"ID":"d42fc83c-2ce5-483b-826c-d66ac55d65d7","Type":"ContainerDied","Data":"daf4579bd1d61d6622ab1bb716ec5e7b397e96cbe75b5a02fad3f5a891946c87"}
Aug 04 18:37:43.117146 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:43.116857    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Aug 04 18:37:53.116251 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:37:53.116197    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Aug 04 18:38:01.498300 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:01.498265    2566 scope.go:117] "RemoveContainer" containerID="ef8f0fb3581e22759495499f7ccb4f24c7bcce0535e7df22c583a6938e8b431d"
Aug 04 18:38:01.508829 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:01.508803    2566 scope.go:117] "RemoveContainer" containerID="c0aeac4fbdba1d589139d88a04c6b0ad4e14da091304e7586d9e140c925f909b"
Aug 04 18:38:01.518547 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:01.518516    2566 scope.go:117] "RemoveContainer" containerID="fdf18a1ce30d7e95d7bf505c86fe9d83ced3c52062e579df0346dbd06e322663"
Aug 04 18:38:03.116741 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:03.116699    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Aug 04 18:38:13.117244 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:13.117191    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:38:21.143237 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:21.143030    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"]
Aug 04 18:38:21.143869 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:21.143324    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container" containerID="cri-o://f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545" gracePeriod=30
Aug 04 18:38:21.143869 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:21.143740    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kube-rbac-proxy" containerID="cri-o://fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399" gracePeriod=30
Aug 04 18:38:23.112041 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:23.111994    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Aug 04 18:38:23.116444 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:23.116408    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Aug 04 18:38:26.375640 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.375506    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"]
Aug 04 18:38:26.397677 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.397647    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"]
Aug 04 18:38:26.397864 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.397785    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:26.400622 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.400543    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Aug 04 18:38:26.400622 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.400564    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Aug 04 18:38:26.458015 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.457956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/302e1b89-32e3-4c00-bc7a-0523b4937770-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-55768c9995-zkd6l\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:26.458192 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.458034    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd27b\" (UniqueName: \"kubernetes.io/projected/302e1b89-32e3-4c00-bc7a-0523b4937770-kube-api-access-qd27b\") pod \"isvc-canary-stop-v2-predictor-55768c9995-zkd6l\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:26.458192 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.458143    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/302e1b89-32e3-4c00-bc7a-0523b4937770-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-55768c9995-zkd6l\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:26.458313 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.458286    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/302e1b89-32e3-4c00-bc7a-0523b4937770-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-55768c9995-zkd6l\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:26.458407 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.458328    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/302e1b89-32e3-4c00-bc7a-0523b4937770-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-55768c9995-zkd6l\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:26.559061 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.559022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/302e1b89-32e3-4c00-bc7a-0523b4937770-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-55768c9995-zkd6l\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:26.559296 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.559101    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/302e1b89-32e3-4c00-bc7a-0523b4937770-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-55768c9995-zkd6l\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:26.559296 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.559122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/302e1b89-32e3-4c00-bc7a-0523b4937770-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-55768c9995-zkd6l\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:26.559296 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.559142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/302e1b89-32e3-4c00-bc7a-0523b4937770-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-55768c9995-zkd6l\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:26.559296 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.559195    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd27b\" (UniqueName: \"kubernetes.io/projected/302e1b89-32e3-4c00-bc7a-0523b4937770-kube-api-access-qd27b\") pod \"isvc-canary-stop-v2-predictor-55768c9995-zkd6l\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:26.559561 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:38:26.559295    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Aug 04 18:38:26.559561 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:38:26.559388    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/302e1b89-32e3-4c00-bc7a-0523b4937770-proxy-tls podName:302e1b89-32e3-4c00-bc7a-0523b4937770 nodeName:}" failed. No retries permitted until 2026-08-04 18:38:27.059344632 +0000 UTC m=+2306.349958787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/302e1b89-32e3-4c00-bc7a-0523b4937770-proxy-tls") pod "isvc-canary-stop-v2-predictor-55768c9995-zkd6l" (UID: "302e1b89-32e3-4c00-bc7a-0523b4937770") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Aug 04 18:38:26.559561 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.559437    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/302e1b89-32e3-4c00-bc7a-0523b4937770-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-55768c9995-zkd6l\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:26.559848 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.559826    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/302e1b89-32e3-4c00-bc7a-0523b4937770-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-55768c9995-zkd6l\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:26.559989 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.559959    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/302e1b89-32e3-4c00-bc7a-0523b4937770-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-55768c9995-zkd6l\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:26.575217 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:26.575181    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd27b\" (UniqueName: \"kubernetes.io/projected/302e1b89-32e3-4c00-bc7a-0523b4937770-kube-api-access-qd27b\") pod \"isvc-canary-stop-v2-predictor-55768c9995-zkd6l\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:27.063773 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:27.063733    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/302e1b89-32e3-4c00-bc7a-0523b4937770-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-55768c9995-zkd6l\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:27.066591 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:27.066561    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/302e1b89-32e3-4c00-bc7a-0523b4937770-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-55768c9995-zkd6l\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:27.310845 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:27.310811    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:38:28.111626 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:28.111577    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Aug 04 18:38:33.111654 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:33.111607    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Aug 04 18:38:33.112257 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:33.111749    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:38:33.116828 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:33.116794    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Aug 04 18:38:38.111404 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:38.111339    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Aug 04 18:38:43.111554 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:43.111512    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Aug 04 18:38:43.116998 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:43.116954    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Aug 04 18:38:43.117161 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:43.117082    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:38:48.112033 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.111984    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Aug 04 18:38:48.692279 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.692249    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:38:48.794712 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.794685    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"]
Aug 04 18:38:48.814094 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.814060    2566 generic.go:356] "Generic (PLEG): container finished" podID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerID="fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399" exitCode=2
Aug 04 18:38:48.814094 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.814090    2566 generic.go:356] "Generic (PLEG): container finished" podID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerID="f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545" exitCode=0
Aug 04 18:38:48.814333 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.814115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" event={"ID":"d0808b6a-104b-4486-a7bd-3eef07f71b7d","Type":"ContainerDied","Data":"fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399"}
Aug 04 18:38:48.814333 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.814134    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"
Aug 04 18:38:48.814333 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.814178    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" event={"ID":"d0808b6a-104b-4486-a7bd-3eef07f71b7d","Type":"ContainerDied","Data":"f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545"}
Aug 04 18:38:48.814333 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.814212    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp" event={"ID":"d0808b6a-104b-4486-a7bd-3eef07f71b7d","Type":"ContainerDied","Data":"51e1c8981f8370078655ec0e35c878c08777ac790b439d699a24506fa9211d48"}
Aug 04 18:38:48.814333 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.814217    2566 scope.go:117] "RemoveContainer" containerID="fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399"
Aug 04 18:38:48.881583 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.881543    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0808b6a-104b-4486-a7bd-3eef07f71b7d-proxy-tls\") pod \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") "
Aug 04 18:38:48.881583 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.881586    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0808b6a-104b-4486-a7bd-3eef07f71b7d-cabundle-cert\") pod \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") "
Aug 04 18:38:48.881838 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.881655    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kserve-provision-location\") pod \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") "
Aug 04 18:38:48.881838 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.881756    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5f59\" (UniqueName: \"kubernetes.io/projected/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kube-api-access-w5f59\") pod \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") "
Aug 04 18:38:48.881838 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.881791    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kube-rbac-proxy-sar-config\") pod \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\" (UID: \"d0808b6a-104b-4486-a7bd-3eef07f71b7d\") "
Aug 04 18:38:48.882049 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.882025    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0808b6a-104b-4486-a7bd-3eef07f71b7d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0808b6a-104b-4486-a7bd-3eef07f71b7d" (UID: "d0808b6a-104b-4486-a7bd-3eef07f71b7d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:38:48.882049 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.882031    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0808b6a-104b-4486-a7bd-3eef07f71b7d" (UID: "d0808b6a-104b-4486-a7bd-3eef07f71b7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:38:48.882397 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.882334    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0808b6a-104b-4486-a7bd-3eef07f71b7d" (UID: "d0808b6a-104b-4486-a7bd-3eef07f71b7d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:38:48.884288 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.884259    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kube-api-access-w5f59" (OuterVolumeSpecName: "kube-api-access-w5f59") pod "d0808b6a-104b-4486-a7bd-3eef07f71b7d" (UID: "d0808b6a-104b-4486-a7bd-3eef07f71b7d"). InnerVolumeSpecName "kube-api-access-w5f59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:38:48.884425 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.884272    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0808b6a-104b-4486-a7bd-3eef07f71b7d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0808b6a-104b-4486-a7bd-3eef07f71b7d" (UID: "d0808b6a-104b-4486-a7bd-3eef07f71b7d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:38:48.982938 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.982847    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:38:48.982938 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.982884    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5f59\" (UniqueName: \"kubernetes.io/projected/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kube-api-access-w5f59\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:38:48.982938 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.982898    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0808b6a-104b-4486-a7bd-3eef07f71b7d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:38:48.982938 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.982911    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0808b6a-104b-4486-a7bd-3eef07f71b7d-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:38:48.982938 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:48.982924    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0808b6a-104b-4486-a7bd-3eef07f71b7d-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:38:49.148179 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:49.148142    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"]
Aug 04 18:38:49.150334 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:49.150306    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-gf8fp"]
Aug 04 18:38:49.195775 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:49.195742    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" path="/var/lib/kubelet/pods/d0808b6a-104b-4486-a7bd-3eef07f71b7d/volumes"
Aug 04 18:38:49.481170 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:38:49.481128    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod302e1b89_32e3_4c00_bc7a_0523b4937770.slice/crio-d9c567fa572d54ad542dd20498b800f4f514988a8f0aabc53dd36ef126ec96a7 WatchSource:0}: Error finding container d9c567fa572d54ad542dd20498b800f4f514988a8f0aabc53dd36ef126ec96a7: Status 404 returned error can't find the container with id d9c567fa572d54ad542dd20498b800f4f514988a8f0aabc53dd36ef126ec96a7
Aug 04 18:38:49.486332 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:49.486312    2566 scope.go:117] "RemoveContainer" containerID="f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545"
Aug 04 18:38:49.822771 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:49.822737    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" event={"ID":"302e1b89-32e3-4c00-bc7a-0523b4937770","Type":"ContainerStarted","Data":"d9c567fa572d54ad542dd20498b800f4f514988a8f0aabc53dd36ef126ec96a7"}
Aug 04 18:38:50.571471 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:50.571450    2566 scope.go:117] "RemoveContainer" containerID="09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c"
Aug 04 18:38:50.668304 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:50.668283    2566 scope.go:117] "RemoveContainer" containerID="fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399"
Aug 04 18:38:50.668636 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:38:50.668617    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399\": container with ID starting with fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399 not found: ID does not exist" containerID="fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399"
Aug 04 18:38:50.668702 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:50.668645    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399"} err="failed to get container status \"fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399\": rpc error: code = NotFound desc = could not find container \"fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399\": container with ID starting with fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399 not found: ID does not exist"
Aug 04 18:38:50.668702 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:50.668663    2566 scope.go:117] "RemoveContainer" containerID="f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545"
Aug 04 18:38:50.668905 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:38:50.668883    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545\": container with ID starting with f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545 not found: ID does not exist" containerID="f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545"
Aug 04 18:38:50.668952 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:50.668912    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545"} err="failed to get container status \"f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545\": rpc error: code = NotFound desc = could not find container \"f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545\": container with ID starting with f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545 not found: ID does not exist"
Aug 04 18:38:50.668952 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:50.668927    2566 scope.go:117] "RemoveContainer" containerID="09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c"
Aug 04 18:38:50.669168 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:38:50.669149    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c\": container with ID starting with 09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c not found: ID does not exist" containerID="09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c"
Aug 04 18:38:50.669246 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:50.669175    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c"} err="failed to get container status \"09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c\": rpc error: code = NotFound desc = could not find container \"09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c\": container with ID starting with 09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c not found: ID does not exist"
Aug 04 18:38:50.669246 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:50.669194    2566 scope.go:117] "RemoveContainer" containerID="fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399"
Aug 04 18:38:50.669442 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:50.669421    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399"} err="failed to get container status \"fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399\": rpc error: code = NotFound desc = could not find container \"fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399\": container with ID starting with fcb1bc9303cc4cfbb97c7bd8c2295bb4d280bf8867d0f472e89454bb4352d399 not found: ID does not exist"
Aug 04 18:38:50.669526 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:50.669443    2566 scope.go:117] "RemoveContainer" containerID="f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545"
Aug 04 18:38:50.669651 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:50.669631    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545"} err="failed to get container status \"f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545\": rpc error: code = NotFound desc = could not find container \"f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545\": container with ID starting with f4ab26c50d4b40ae1dc8e0a7bf1a4924351e7cda7cae7f0ec3ccc91161074545 not found: ID does not exist"
Aug 04 18:38:50.669705 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:50.669653    2566 scope.go:117] "RemoveContainer" containerID="09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c"
Aug 04 18:38:50.669873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:38:50.669856    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c"} err="failed to get container status \"09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c\": rpc error: code = NotFound desc = could not find container \"09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c\": container with ID starting with 09c6b33c40320ee216aeccde0e60fa4f51ccd0b7821b2beff41e65cd2ed7242c not found: ID does not exist"
Aug 04 18:39:00.889067 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:00.888971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" event={"ID":"302e1b89-32e3-4c00-bc7a-0523b4937770","Type":"ContainerStarted","Data":"3393be011871c86230f13afe0464e526a4eeed5bf04e512f62d63a9ce9bdc1e7"}
Aug 04 18:39:29.044757 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:29.044722    2566 generic.go:356] "Generic (PLEG): container finished" podID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerID="3393be011871c86230f13afe0464e526a4eeed5bf04e512f62d63a9ce9bdc1e7" exitCode=0
Aug 04 18:39:29.045158 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:29.044790    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" event={"ID":"302e1b89-32e3-4c00-bc7a-0523b4937770","Type":"ContainerDied","Data":"3393be011871c86230f13afe0464e526a4eeed5bf04e512f62d63a9ce9bdc1e7"}
Aug 04 18:39:30.054180 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:30.054148    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" event={"ID":"302e1b89-32e3-4c00-bc7a-0523b4937770","Type":"ContainerStarted","Data":"3b6f8f99bed9bd832229484005c88498c6fc8cf20b4f6addd4af06ee215276d4"}
Aug 04 18:39:30.054580 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:30.054227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" event={"ID":"302e1b89-32e3-4c00-bc7a-0523b4937770","Type":"ContainerStarted","Data":"2c2645599f638be8b08fd831b986adb39b02359acf4f22d6f8778f9ec275e60f"}
Aug 04 18:39:30.054580 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:30.054488    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:39:30.078475 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:30.078431    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" podStartSLOduration=60.195623022 podStartE2EDuration="1m4.078414846s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.882791824s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:39:30.076026273 +0000 UTC m=+2369.366640465" watchObservedRunningTime="2026-08-04 18:39:30.078414846 +0000 UTC m=+2369.369029017"
Aug 04 18:39:31.062575 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:31.062541    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr" event={"ID":"d42fc83c-2ce5-483b-826c-d66ac55d65d7","Type":"ContainerStarted","Data":"93836798f410a1fdc01853884e40b13be7e3520a7d36bb6374b926834ee9fe9a"}
Aug 04 18:39:31.063076 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:31.062600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr" event={"ID":"d42fc83c-2ce5-483b-826c-d66ac55d65d7","Type":"ContainerStarted","Data":"9d4e4a280b8aa27e2b945ef8d24332dcf374b1259153303173e49c8c46bbbbad"}
Aug 04 18:39:31.063076 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:31.062637    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:39:31.063076 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:31.062685    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:39:31.063222 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:31.063101    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:39:31.063811 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:31.063785    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr" podUID="d42fc83c-2ce5-483b-826c-d66ac55d65d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Aug 04 18:39:31.063910 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:31.063867    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Aug 04 18:39:31.087236 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:31.087186    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr" podStartSLOduration=2.475460922 podStartE2EDuration="2m1.087167195s" totalImagesPullingTime="1m54.661300815s" totalInitContainerRuntime="3.950405458s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:39:31.086923051 +0000 UTC m=+2370.377537238" watchObservedRunningTime="2026-08-04 18:39:31.087167195 +0000 UTC m=+2370.377781367"
Aug 04 18:39:32.067517 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:32.067474    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Aug 04 18:39:32.067927 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:32.067580    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr" podUID="d42fc83c-2ce5-483b-826c-d66ac55d65d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Aug 04 18:39:37.074816 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:37.074786    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:39:37.075522 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:37.075173    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:39:37.075522 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:37.075209    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Aug 04 18:39:37.075896 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:37.075877    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:39:42.993442 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:42.993408    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"]
Aug 04 18:39:42.994010 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:42.993731    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr" podUID="d42fc83c-2ce5-483b-826c-d66ac55d65d7" containerName="kube-rbac-proxy" containerID="cri-o://93836798f410a1fdc01853884e40b13be7e3520a7d36bb6374b926834ee9fe9a" gracePeriod=30
Aug 04 18:39:42.994010 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:42.993712    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr" podUID="d42fc83c-2ce5-483b-826c-d66ac55d65d7" containerName="kserve-container" containerID="cri-o://9d4e4a280b8aa27e2b945ef8d24332dcf374b1259153303173e49c8c46bbbbad" gracePeriod=30
Aug 04 18:39:43.120544 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:43.120509    2566 generic.go:356] "Generic (PLEG): container finished" podID="d42fc83c-2ce5-483b-826c-d66ac55d65d7" containerID="93836798f410a1fdc01853884e40b13be7e3520a7d36bb6374b926834ee9fe9a" exitCode=2
Aug 04 18:39:43.120717 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:43.120583    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr" event={"ID":"d42fc83c-2ce5-483b-826c-d66ac55d65d7","Type":"ContainerDied","Data":"93836798f410a1fdc01853884e40b13be7e3520a7d36bb6374b926834ee9fe9a"}
Aug 04 18:39:45.967240 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:45.967205    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:39:46.024183 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.024090    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kserve-provision-location\") pod \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") "
Aug 04 18:39:46.024183 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.024139    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6v9t\" (UniqueName: \"kubernetes.io/projected/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kube-api-access-b6v9t\") pod \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") "
Aug 04 18:39:46.024494 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.024188    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d42fc83c-2ce5-483b-826c-d66ac55d65d7-cabundle-cert\") pod \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") "
Aug 04 18:39:46.024494 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.024236    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d42fc83c-2ce5-483b-826c-d66ac55d65d7-proxy-tls\") pod \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") "
Aug 04 18:39:46.024494 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.024260    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kube-rbac-proxy-sar-config\") pod \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\" (UID: \"d42fc83c-2ce5-483b-826c-d66ac55d65d7\") "
Aug 04 18:39:46.024650 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.024599    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d42fc83c-2ce5-483b-826c-d66ac55d65d7" (UID: "d42fc83c-2ce5-483b-826c-d66ac55d65d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:39:46.024650 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.024618    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42fc83c-2ce5-483b-826c-d66ac55d65d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d42fc83c-2ce5-483b-826c-d66ac55d65d7" (UID: "d42fc83c-2ce5-483b-826c-d66ac55d65d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:39:46.024650 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.024620    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d42fc83c-2ce5-483b-826c-d66ac55d65d7" (UID: "d42fc83c-2ce5-483b-826c-d66ac55d65d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:39:46.026361 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.026325    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d42fc83c-2ce5-483b-826c-d66ac55d65d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d42fc83c-2ce5-483b-826c-d66ac55d65d7" (UID: "d42fc83c-2ce5-483b-826c-d66ac55d65d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:39:46.026427 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.026373    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kube-api-access-b6v9t" (OuterVolumeSpecName: "kube-api-access-b6v9t") pod "d42fc83c-2ce5-483b-826c-d66ac55d65d7" (UID: "d42fc83c-2ce5-483b-826c-d66ac55d65d7"). InnerVolumeSpecName "kube-api-access-b6v9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:39:46.125269 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.125238    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:39:46.125269 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.125265    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6v9t\" (UniqueName: \"kubernetes.io/projected/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kube-api-access-b6v9t\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:39:46.125269 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.125275    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d42fc83c-2ce5-483b-826c-d66ac55d65d7-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:39:46.125545 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.125287    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d42fc83c-2ce5-483b-826c-d66ac55d65d7-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:39:46.125545 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.125295    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d42fc83c-2ce5-483b-826c-d66ac55d65d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:39:46.136484 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.136457    2566 generic.go:356] "Generic (PLEG): container finished" podID="d42fc83c-2ce5-483b-826c-d66ac55d65d7" containerID="9d4e4a280b8aa27e2b945ef8d24332dcf374b1259153303173e49c8c46bbbbad" exitCode=0
Aug 04 18:39:46.136642 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.136516    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"
Aug 04 18:39:46.136642 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.136549    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr" event={"ID":"d42fc83c-2ce5-483b-826c-d66ac55d65d7","Type":"ContainerDied","Data":"9d4e4a280b8aa27e2b945ef8d24332dcf374b1259153303173e49c8c46bbbbad"}
Aug 04 18:39:46.136642 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.136604    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr" event={"ID":"d42fc83c-2ce5-483b-826c-d66ac55d65d7","Type":"ContainerDied","Data":"b3ed036de45ecbf0646d242f25081e80166ab2763ca7cf47fdc7c26c6b82b4d4"}
Aug 04 18:39:46.136642 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.136629    2566 scope.go:117] "RemoveContainer" containerID="93836798f410a1fdc01853884e40b13be7e3520a7d36bb6374b926834ee9fe9a"
Aug 04 18:39:46.145250 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.145226    2566 scope.go:117] "RemoveContainer" containerID="9d4e4a280b8aa27e2b945ef8d24332dcf374b1259153303173e49c8c46bbbbad"
Aug 04 18:39:46.155154 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.155133    2566 scope.go:117] "RemoveContainer" containerID="daf4579bd1d61d6622ab1bb716ec5e7b397e96cbe75b5a02fad3f5a891946c87"
Aug 04 18:39:46.162211 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.162183    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"]
Aug 04 18:39:46.164011 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.163996    2566 scope.go:117] "RemoveContainer" containerID="93836798f410a1fdc01853884e40b13be7e3520a7d36bb6374b926834ee9fe9a"
Aug 04 18:39:46.164288 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:39:46.164271    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93836798f410a1fdc01853884e40b13be7e3520a7d36bb6374b926834ee9fe9a\": container with ID starting with 93836798f410a1fdc01853884e40b13be7e3520a7d36bb6374b926834ee9fe9a not found: ID does not exist" containerID="93836798f410a1fdc01853884e40b13be7e3520a7d36bb6374b926834ee9fe9a"
Aug 04 18:39:46.164336 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.164295    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93836798f410a1fdc01853884e40b13be7e3520a7d36bb6374b926834ee9fe9a"} err="failed to get container status \"93836798f410a1fdc01853884e40b13be7e3520a7d36bb6374b926834ee9fe9a\": rpc error: code = NotFound desc = could not find container \"93836798f410a1fdc01853884e40b13be7e3520a7d36bb6374b926834ee9fe9a\": container with ID starting with 93836798f410a1fdc01853884e40b13be7e3520a7d36bb6374b926834ee9fe9a not found: ID does not exist"
Aug 04 18:39:46.164336 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.164312    2566 scope.go:117] "RemoveContainer" containerID="9d4e4a280b8aa27e2b945ef8d24332dcf374b1259153303173e49c8c46bbbbad"
Aug 04 18:39:46.164557 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:39:46.164542    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d4e4a280b8aa27e2b945ef8d24332dcf374b1259153303173e49c8c46bbbbad\": container with ID starting with 9d4e4a280b8aa27e2b945ef8d24332dcf374b1259153303173e49c8c46bbbbad not found: ID does not exist" containerID="9d4e4a280b8aa27e2b945ef8d24332dcf374b1259153303173e49c8c46bbbbad"
Aug 04 18:39:46.164600 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.164562    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4e4a280b8aa27e2b945ef8d24332dcf374b1259153303173e49c8c46bbbbad"} err="failed to get container status \"9d4e4a280b8aa27e2b945ef8d24332dcf374b1259153303173e49c8c46bbbbad\": rpc error: code = NotFound desc = could not find container \"9d4e4a280b8aa27e2b945ef8d24332dcf374b1259153303173e49c8c46bbbbad\": container with ID starting with 9d4e4a280b8aa27e2b945ef8d24332dcf374b1259153303173e49c8c46bbbbad not found: ID does not exist"
Aug 04 18:39:46.164600 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.164577    2566 scope.go:117] "RemoveContainer" containerID="daf4579bd1d61d6622ab1bb716ec5e7b397e96cbe75b5a02fad3f5a891946c87"
Aug 04 18:39:46.164790 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:39:46.164775    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf4579bd1d61d6622ab1bb716ec5e7b397e96cbe75b5a02fad3f5a891946c87\": container with ID starting with daf4579bd1d61d6622ab1bb716ec5e7b397e96cbe75b5a02fad3f5a891946c87 not found: ID does not exist" containerID="daf4579bd1d61d6622ab1bb716ec5e7b397e96cbe75b5a02fad3f5a891946c87"
Aug 04 18:39:46.164874 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.164793    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf4579bd1d61d6622ab1bb716ec5e7b397e96cbe75b5a02fad3f5a891946c87"} err="failed to get container status \"daf4579bd1d61d6622ab1bb716ec5e7b397e96cbe75b5a02fad3f5a891946c87\": rpc error: code = NotFound desc = could not find container \"daf4579bd1d61d6622ab1bb716ec5e7b397e96cbe75b5a02fad3f5a891946c87\": container with ID starting with daf4579bd1d61d6622ab1bb716ec5e7b397e96cbe75b5a02fad3f5a891946c87 not found: ID does not exist"
Aug 04 18:39:46.170041 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:46.170017    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-59slr"]
Aug 04 18:39:47.075455 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:47.075421    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Aug 04 18:39:47.194325 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:47.194295    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d42fc83c-2ce5-483b-826c-d66ac55d65d7" path="/var/lib/kubelet/pods/d42fc83c-2ce5-483b-826c-d66ac55d65d7/volumes"
Aug 04 18:39:48.155902 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.155849    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"]
Aug 04 18:39:48.156348 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.156262    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d42fc83c-2ce5-483b-826c-d66ac55d65d7" containerName="kube-rbac-proxy"
Aug 04 18:39:48.156348 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.156277    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d42fc83c-2ce5-483b-826c-d66ac55d65d7" containerName="kube-rbac-proxy"
Aug 04 18:39:48.156348 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.156296    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="storage-initializer"
Aug 04 18:39:48.156348 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.156302    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="storage-initializer"
Aug 04 18:39:48.156348 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.156310    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container"
Aug 04 18:39:48.156348 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.156315    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container"
Aug 04 18:39:48.156348 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.156326    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d42fc83c-2ce5-483b-826c-d66ac55d65d7" containerName="storage-initializer"
Aug 04 18:39:48.156348 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.156334    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d42fc83c-2ce5-483b-826c-d66ac55d65d7" containerName="storage-initializer"
Aug 04 18:39:48.156348 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.156373    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kube-rbac-proxy"
Aug 04 18:39:48.156348 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.156379    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kube-rbac-proxy"
Aug 04 18:39:48.156739 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.156388    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d42fc83c-2ce5-483b-826c-d66ac55d65d7" containerName="kserve-container"
Aug 04 18:39:48.156739 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.156403    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d42fc83c-2ce5-483b-826c-d66ac55d65d7" containerName="kserve-container"
Aug 04 18:39:48.156739 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.156466    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d42fc83c-2ce5-483b-826c-d66ac55d65d7" containerName="kserve-container"
Aug 04 18:39:48.156739 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.156474    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kube-rbac-proxy"
Aug 04 18:39:48.156739 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.156482    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d42fc83c-2ce5-483b-826c-d66ac55d65d7" containerName="kube-rbac-proxy"
Aug 04 18:39:48.156739 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.156488    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0808b6a-104b-4486-a7bd-3eef07f71b7d" containerName="kserve-container"
Aug 04 18:39:48.232454 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.232419    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"]
Aug 04 18:39:48.232640 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.232566    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.235033 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.235009    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Aug 04 18:39:48.235182 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.235093    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Aug 04 18:39:48.345299 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.345267    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/549308d8-e567-4ded-bba3-2adfd6a70dce-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-jsrtd\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.345535 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.345308    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s682k\" (UniqueName: \"kubernetes.io/projected/549308d8-e567-4ded-bba3-2adfd6a70dce-kube-api-access-s682k\") pod \"isvc-xgboost-predictor-7566d65bf7-jsrtd\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.345535 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.345336    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/549308d8-e567-4ded-bba3-2adfd6a70dce-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-jsrtd\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.345535 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.345492    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/549308d8-e567-4ded-bba3-2adfd6a70dce-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-jsrtd\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.345704 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.345682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/549308d8-e567-4ded-bba3-2adfd6a70dce-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-jsrtd\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.446154 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.446072    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/549308d8-e567-4ded-bba3-2adfd6a70dce-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-jsrtd\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.446154 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.446132    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/549308d8-e567-4ded-bba3-2adfd6a70dce-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-jsrtd\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.446408 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.446262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s682k\" (UniqueName: \"kubernetes.io/projected/549308d8-e567-4ded-bba3-2adfd6a70dce-kube-api-access-s682k\") pod \"isvc-xgboost-predictor-7566d65bf7-jsrtd\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.446408 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.446321    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/549308d8-e567-4ded-bba3-2adfd6a70dce-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-jsrtd\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.446527 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.446405    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/549308d8-e567-4ded-bba3-2adfd6a70dce-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-jsrtd\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.446604 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.446583    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/549308d8-e567-4ded-bba3-2adfd6a70dce-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-jsrtd\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.447048 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.447025    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/549308d8-e567-4ded-bba3-2adfd6a70dce-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-jsrtd\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.447111 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.447095    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/549308d8-e567-4ded-bba3-2adfd6a70dce-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-jsrtd\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.448837 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.448816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/549308d8-e567-4ded-bba3-2adfd6a70dce-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-jsrtd\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.456585 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.456564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s682k\" (UniqueName: \"kubernetes.io/projected/549308d8-e567-4ded-bba3-2adfd6a70dce-kube-api-access-s682k\") pod \"isvc-xgboost-predictor-7566d65bf7-jsrtd\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.542959 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.542921    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:39:48.714865 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:48.714780    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"]
Aug 04 18:39:48.719057 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:39:48.719026    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549308d8_e567_4ded_bba3_2adfd6a70dce.slice/crio-477c74d4ba1154542e7e0c8c250f3b6079723e24584dd3d8997a5cdebc905ad3 WatchSource:0}: Error finding container 477c74d4ba1154542e7e0c8c250f3b6079723e24584dd3d8997a5cdebc905ad3: Status 404 returned error can't find the container with id 477c74d4ba1154542e7e0c8c250f3b6079723e24584dd3d8997a5cdebc905ad3
Aug 04 18:39:49.155963 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:49.155928    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" event={"ID":"549308d8-e567-4ded-bba3-2adfd6a70dce","Type":"ContainerStarted","Data":"bcfaf9a7533eefcbbc214cc78218112ae8d40a2514240607c74c9678aa8817d6"}
Aug 04 18:39:49.156520 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:49.156000    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" event={"ID":"549308d8-e567-4ded-bba3-2adfd6a70dce","Type":"ContainerStarted","Data":"477c74d4ba1154542e7e0c8c250f3b6079723e24584dd3d8997a5cdebc905ad3"}
Aug 04 18:39:53.178318 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:53.178285    2566 generic.go:356] "Generic (PLEG): container finished" podID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerID="bcfaf9a7533eefcbbc214cc78218112ae8d40a2514240607c74c9678aa8817d6" exitCode=0
Aug 04 18:39:53.178732 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:53.178376    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" event={"ID":"549308d8-e567-4ded-bba3-2adfd6a70dce","Type":"ContainerDied","Data":"bcfaf9a7533eefcbbc214cc78218112ae8d40a2514240607c74c9678aa8817d6"}
Aug 04 18:39:57.075543 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:39:57.075501    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Aug 04 18:40:07.075941 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:07.075898    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Aug 04 18:40:15.305668 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:15.305634    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" event={"ID":"549308d8-e567-4ded-bba3-2adfd6a70dce","Type":"ContainerStarted","Data":"4ead25a0a8c3fae177b52dfcf7bae36d5641574705646b486177b59f9497da4d"}
Aug 04 18:40:15.306194 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:15.305705    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" event={"ID":"549308d8-e567-4ded-bba3-2adfd6a70dce","Type":"ContainerStarted","Data":"58953d7cd0610725f58da97c367b86229bb82ab222f207065480ca379fbc25f1"}
Aug 04 18:40:15.306194 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:15.305942    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:40:15.331654 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:15.331594    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" podStartSLOduration=2.496973129 podStartE2EDuration="27.33157219s" totalImagesPullingTime="21.22030345s" totalInitContainerRuntime="3.614295611s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:39:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:40:15.329933891 +0000 UTC m=+2414.620548093" watchObservedRunningTime="2026-08-04 18:40:15.33157219 +0000 UTC m=+2414.622186362"
Aug 04 18:40:16.310089 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:16.310059    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:40:16.311371 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:16.311323    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Aug 04 18:40:17.076264 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:17.076221    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Aug 04 18:40:17.315189 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:17.315141    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Aug 04 18:40:22.319013 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:22.318987    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:40:22.319617 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:22.319497    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Aug 04 18:40:27.075799 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:27.075717    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Aug 04 18:40:32.319698 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:32.319657    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Aug 04 18:40:37.075852 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:37.075808    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Aug 04 18:40:42.319584 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:42.319535    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Aug 04 18:40:47.075876 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:47.075826    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Aug 04 18:40:49.194723 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:49.194696    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:40:51.443004 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:51.442963    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"]
Aug 04 18:40:51.443436 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:51.443268    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kserve-container" containerID="cri-o://2c2645599f638be8b08fd831b986adb39b02359acf4f22d6f8778f9ec275e60f" gracePeriod=30
Aug 04 18:40:51.443436 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:51.443300    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kube-rbac-proxy" containerID="cri-o://3b6f8f99bed9bd832229484005c88498c6fc8cf20b4f6addd4af06ee215276d4" gracePeriod=30
Aug 04 18:40:52.068131 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:52.068088    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Aug 04 18:40:52.319544 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:52.319459    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Aug 04 18:40:52.487201 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:52.487167    2566 generic.go:356] "Generic (PLEG): container finished" podID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerID="3b6f8f99bed9bd832229484005c88498c6fc8cf20b4f6addd4af06ee215276d4" exitCode=2
Aug 04 18:40:52.487596 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:52.487237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" event={"ID":"302e1b89-32e3-4c00-bc7a-0523b4937770","Type":"ContainerDied","Data":"3b6f8f99bed9bd832229484005c88498c6fc8cf20b4f6addd4af06ee215276d4"}
Aug 04 18:40:56.388746 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.388723    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:40:56.508693 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.508608    2566 generic.go:356] "Generic (PLEG): container finished" podID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerID="2c2645599f638be8b08fd831b986adb39b02359acf4f22d6f8778f9ec275e60f" exitCode=0
Aug 04 18:40:56.508693 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.508666    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"
Aug 04 18:40:56.508933 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.508704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" event={"ID":"302e1b89-32e3-4c00-bc7a-0523b4937770","Type":"ContainerDied","Data":"2c2645599f638be8b08fd831b986adb39b02359acf4f22d6f8778f9ec275e60f"}
Aug 04 18:40:56.508933 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.508777    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l" event={"ID":"302e1b89-32e3-4c00-bc7a-0523b4937770","Type":"ContainerDied","Data":"d9c567fa572d54ad542dd20498b800f4f514988a8f0aabc53dd36ef126ec96a7"}
Aug 04 18:40:56.508933 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.508814    2566 scope.go:117] "RemoveContainer" containerID="3b6f8f99bed9bd832229484005c88498c6fc8cf20b4f6addd4af06ee215276d4"
Aug 04 18:40:56.517253 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.517234    2566 scope.go:117] "RemoveContainer" containerID="2c2645599f638be8b08fd831b986adb39b02359acf4f22d6f8778f9ec275e60f"
Aug 04 18:40:56.524800 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.524785    2566 scope.go:117] "RemoveContainer" containerID="3393be011871c86230f13afe0464e526a4eeed5bf04e512f62d63a9ce9bdc1e7"
Aug 04 18:40:56.532326 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.532312    2566 scope.go:117] "RemoveContainer" containerID="3b6f8f99bed9bd832229484005c88498c6fc8cf20b4f6addd4af06ee215276d4"
Aug 04 18:40:56.532648 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:40:56.532624    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6f8f99bed9bd832229484005c88498c6fc8cf20b4f6addd4af06ee215276d4\": container with ID starting with 3b6f8f99bed9bd832229484005c88498c6fc8cf20b4f6addd4af06ee215276d4 not found: ID does not exist" containerID="3b6f8f99bed9bd832229484005c88498c6fc8cf20b4f6addd4af06ee215276d4"
Aug 04 18:40:56.532743 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.532656    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6f8f99bed9bd832229484005c88498c6fc8cf20b4f6addd4af06ee215276d4"} err="failed to get container status \"3b6f8f99bed9bd832229484005c88498c6fc8cf20b4f6addd4af06ee215276d4\": rpc error: code = NotFound desc = could not find container \"3b6f8f99bed9bd832229484005c88498c6fc8cf20b4f6addd4af06ee215276d4\": container with ID starting with 3b6f8f99bed9bd832229484005c88498c6fc8cf20b4f6addd4af06ee215276d4 not found: ID does not exist"
Aug 04 18:40:56.532743 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.532676    2566 scope.go:117] "RemoveContainer" containerID="2c2645599f638be8b08fd831b986adb39b02359acf4f22d6f8778f9ec275e60f"
Aug 04 18:40:56.532921 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:40:56.532905    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2645599f638be8b08fd831b986adb39b02359acf4f22d6f8778f9ec275e60f\": container with ID starting with 2c2645599f638be8b08fd831b986adb39b02359acf4f22d6f8778f9ec275e60f not found: ID does not exist" containerID="2c2645599f638be8b08fd831b986adb39b02359acf4f22d6f8778f9ec275e60f"
Aug 04 18:40:56.532968 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.532928    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2645599f638be8b08fd831b986adb39b02359acf4f22d6f8778f9ec275e60f"} err="failed to get container status \"2c2645599f638be8b08fd831b986adb39b02359acf4f22d6f8778f9ec275e60f\": rpc error: code = NotFound desc = could not find container \"2c2645599f638be8b08fd831b986adb39b02359acf4f22d6f8778f9ec275e60f\": container with ID starting with 2c2645599f638be8b08fd831b986adb39b02359acf4f22d6f8778f9ec275e60f not found: ID does not exist"
Aug 04 18:40:56.532968 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.532943    2566 scope.go:117] "RemoveContainer" containerID="3393be011871c86230f13afe0464e526a4eeed5bf04e512f62d63a9ce9bdc1e7"
Aug 04 18:40:56.533188 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:40:56.533168    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3393be011871c86230f13afe0464e526a4eeed5bf04e512f62d63a9ce9bdc1e7\": container with ID starting with 3393be011871c86230f13afe0464e526a4eeed5bf04e512f62d63a9ce9bdc1e7 not found: ID does not exist" containerID="3393be011871c86230f13afe0464e526a4eeed5bf04e512f62d63a9ce9bdc1e7"
Aug 04 18:40:56.533249 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.533197    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3393be011871c86230f13afe0464e526a4eeed5bf04e512f62d63a9ce9bdc1e7"} err="failed to get container status \"3393be011871c86230f13afe0464e526a4eeed5bf04e512f62d63a9ce9bdc1e7\": rpc error: code = NotFound desc = could not find container \"3393be011871c86230f13afe0464e526a4eeed5bf04e512f62d63a9ce9bdc1e7\": container with ID starting with 3393be011871c86230f13afe0464e526a4eeed5bf04e512f62d63a9ce9bdc1e7 not found: ID does not exist"
Aug 04 18:40:56.571462 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.571428    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/302e1b89-32e3-4c00-bc7a-0523b4937770-kserve-provision-location\") pod \"302e1b89-32e3-4c00-bc7a-0523b4937770\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") "
Aug 04 18:40:56.571639 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.571479    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/302e1b89-32e3-4c00-bc7a-0523b4937770-kube-rbac-proxy-sar-config\") pod \"302e1b89-32e3-4c00-bc7a-0523b4937770\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") "
Aug 04 18:40:56.571639 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.571525    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qd27b\" (UniqueName: \"kubernetes.io/projected/302e1b89-32e3-4c00-bc7a-0523b4937770-kube-api-access-qd27b\") pod \"302e1b89-32e3-4c00-bc7a-0523b4937770\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") "
Aug 04 18:40:56.571639 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.571565    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/302e1b89-32e3-4c00-bc7a-0523b4937770-cabundle-cert\") pod \"302e1b89-32e3-4c00-bc7a-0523b4937770\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") "
Aug 04 18:40:56.571639 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.571587    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/302e1b89-32e3-4c00-bc7a-0523b4937770-proxy-tls\") pod \"302e1b89-32e3-4c00-bc7a-0523b4937770\" (UID: \"302e1b89-32e3-4c00-bc7a-0523b4937770\") "
Aug 04 18:40:56.571872 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.571825    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/302e1b89-32e3-4c00-bc7a-0523b4937770-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "302e1b89-32e3-4c00-bc7a-0523b4937770" (UID: "302e1b89-32e3-4c00-bc7a-0523b4937770"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:40:56.571958 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.571934    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/302e1b89-32e3-4c00-bc7a-0523b4937770-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "302e1b89-32e3-4c00-bc7a-0523b4937770" (UID: "302e1b89-32e3-4c00-bc7a-0523b4937770"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:40:56.572000 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.571938    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/302e1b89-32e3-4c00-bc7a-0523b4937770-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "302e1b89-32e3-4c00-bc7a-0523b4937770" (UID: "302e1b89-32e3-4c00-bc7a-0523b4937770"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:40:56.573722 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.573700    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302e1b89-32e3-4c00-bc7a-0523b4937770-kube-api-access-qd27b" (OuterVolumeSpecName: "kube-api-access-qd27b") pod "302e1b89-32e3-4c00-bc7a-0523b4937770" (UID: "302e1b89-32e3-4c00-bc7a-0523b4937770"). InnerVolumeSpecName "kube-api-access-qd27b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:40:56.573802 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.573731    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/302e1b89-32e3-4c00-bc7a-0523b4937770-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "302e1b89-32e3-4c00-bc7a-0523b4937770" (UID: "302e1b89-32e3-4c00-bc7a-0523b4937770"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:40:56.672725 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.672695    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/302e1b89-32e3-4c00-bc7a-0523b4937770-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:40:56.672725 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.672720    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/302e1b89-32e3-4c00-bc7a-0523b4937770-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:40:56.672725 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.672730    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd27b\" (UniqueName: \"kubernetes.io/projected/302e1b89-32e3-4c00-bc7a-0523b4937770-kube-api-access-qd27b\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:40:56.672951 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.672740    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/302e1b89-32e3-4c00-bc7a-0523b4937770-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:40:56.672951 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.672750    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/302e1b89-32e3-4c00-bc7a-0523b4937770-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:40:56.832757 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.832723    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"]
Aug 04 18:40:56.841773 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:56.841746    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-zkd6l"]
Aug 04 18:40:57.200959 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:40:57.200872    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" path="/var/lib/kubelet/pods/302e1b89-32e3-4c00-bc7a-0523b4937770/volumes"
Aug 04 18:41:02.319946 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:02.319895    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Aug 04 18:41:12.319954 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:12.319904    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Aug 04 18:41:22.320517 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:22.320487    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:41:28.401782 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:28.401747    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"]
Aug 04 18:41:28.402243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:28.401989    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="kserve-container" containerID="cri-o://58953d7cd0610725f58da97c367b86229bb82ab222f207065480ca379fbc25f1" gracePeriod=30
Aug 04 18:41:28.402243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:28.402042    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="kube-rbac-proxy" containerID="cri-o://4ead25a0a8c3fae177b52dfcf7bae36d5641574705646b486177b59f9497da4d" gracePeriod=30
Aug 04 18:41:28.663947 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:28.663862    2566 generic.go:356] "Generic (PLEG): container finished" podID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerID="4ead25a0a8c3fae177b52dfcf7bae36d5641574705646b486177b59f9497da4d" exitCode=2
Aug 04 18:41:28.663947 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:28.663934    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" event={"ID":"549308d8-e567-4ded-bba3-2adfd6a70dce","Type":"ContainerDied","Data":"4ead25a0a8c3fae177b52dfcf7bae36d5641574705646b486177b59f9497da4d"}
Aug 04 18:41:32.142956 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.142930    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:41:32.283058 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.282961    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/549308d8-e567-4ded-bba3-2adfd6a70dce-proxy-tls\") pod \"549308d8-e567-4ded-bba3-2adfd6a70dce\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") "
Aug 04 18:41:32.283058 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.283043    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/549308d8-e567-4ded-bba3-2adfd6a70dce-cabundle-cert\") pod \"549308d8-e567-4ded-bba3-2adfd6a70dce\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") "
Aug 04 18:41:32.283265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.283100    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/549308d8-e567-4ded-bba3-2adfd6a70dce-kube-rbac-proxy-sar-config\") pod \"549308d8-e567-4ded-bba3-2adfd6a70dce\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") "
Aug 04 18:41:32.283265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.283140    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s682k\" (UniqueName: \"kubernetes.io/projected/549308d8-e567-4ded-bba3-2adfd6a70dce-kube-api-access-s682k\") pod \"549308d8-e567-4ded-bba3-2adfd6a70dce\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") "
Aug 04 18:41:32.283265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.283201    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/549308d8-e567-4ded-bba3-2adfd6a70dce-kserve-provision-location\") pod \"549308d8-e567-4ded-bba3-2adfd6a70dce\" (UID: \"549308d8-e567-4ded-bba3-2adfd6a70dce\") "
Aug 04 18:41:32.283486 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.283457    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549308d8-e567-4ded-bba3-2adfd6a70dce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "549308d8-e567-4ded-bba3-2adfd6a70dce" (UID: "549308d8-e567-4ded-bba3-2adfd6a70dce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:41:32.283594 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.283489    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549308d8-e567-4ded-bba3-2adfd6a70dce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "549308d8-e567-4ded-bba3-2adfd6a70dce" (UID: "549308d8-e567-4ded-bba3-2adfd6a70dce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:41:32.283666 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.283604    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549308d8-e567-4ded-bba3-2adfd6a70dce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "549308d8-e567-4ded-bba3-2adfd6a70dce" (UID: "549308d8-e567-4ded-bba3-2adfd6a70dce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:41:32.285175 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.285159    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549308d8-e567-4ded-bba3-2adfd6a70dce-kube-api-access-s682k" (OuterVolumeSpecName: "kube-api-access-s682k") pod "549308d8-e567-4ded-bba3-2adfd6a70dce" (UID: "549308d8-e567-4ded-bba3-2adfd6a70dce"). InnerVolumeSpecName "kube-api-access-s682k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:41:32.285279 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.285261    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/549308d8-e567-4ded-bba3-2adfd6a70dce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "549308d8-e567-4ded-bba3-2adfd6a70dce" (UID: "549308d8-e567-4ded-bba3-2adfd6a70dce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:41:32.384101 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.384049    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/549308d8-e567-4ded-bba3-2adfd6a70dce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:41:32.384101 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.384094    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s682k\" (UniqueName: \"kubernetes.io/projected/549308d8-e567-4ded-bba3-2adfd6a70dce-kube-api-access-s682k\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:41:32.384101 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.384107    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/549308d8-e567-4ded-bba3-2adfd6a70dce-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:41:32.384398 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.384120    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/549308d8-e567-4ded-bba3-2adfd6a70dce-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:41:32.384398 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.384133    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/549308d8-e567-4ded-bba3-2adfd6a70dce-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:41:32.683362 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.683318    2566 generic.go:356] "Generic (PLEG): container finished" podID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerID="58953d7cd0610725f58da97c367b86229bb82ab222f207065480ca379fbc25f1" exitCode=0
Aug 04 18:41:32.683583 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.683392    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"
Aug 04 18:41:32.683583 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.683409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" event={"ID":"549308d8-e567-4ded-bba3-2adfd6a70dce","Type":"ContainerDied","Data":"58953d7cd0610725f58da97c367b86229bb82ab222f207065480ca379fbc25f1"}
Aug 04 18:41:32.683583 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.683485    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd" event={"ID":"549308d8-e567-4ded-bba3-2adfd6a70dce","Type":"ContainerDied","Data":"477c74d4ba1154542e7e0c8c250f3b6079723e24584dd3d8997a5cdebc905ad3"}
Aug 04 18:41:32.683583 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.683527    2566 scope.go:117] "RemoveContainer" containerID="4ead25a0a8c3fae177b52dfcf7bae36d5641574705646b486177b59f9497da4d"
Aug 04 18:41:32.692615 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.692598    2566 scope.go:117] "RemoveContainer" containerID="58953d7cd0610725f58da97c367b86229bb82ab222f207065480ca379fbc25f1"
Aug 04 18:41:32.700333 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.700311    2566 scope.go:117] "RemoveContainer" containerID="bcfaf9a7533eefcbbc214cc78218112ae8d40a2514240607c74c9678aa8817d6"
Aug 04 18:41:32.708774 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.708752    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"]
Aug 04 18:41:32.709104 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.709081    2566 scope.go:117] "RemoveContainer" containerID="4ead25a0a8c3fae177b52dfcf7bae36d5641574705646b486177b59f9497da4d"
Aug 04 18:41:32.709456 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:41:32.709436    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ead25a0a8c3fae177b52dfcf7bae36d5641574705646b486177b59f9497da4d\": container with ID starting with 4ead25a0a8c3fae177b52dfcf7bae36d5641574705646b486177b59f9497da4d not found: ID does not exist" containerID="4ead25a0a8c3fae177b52dfcf7bae36d5641574705646b486177b59f9497da4d"
Aug 04 18:41:32.709553 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.709470    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ead25a0a8c3fae177b52dfcf7bae36d5641574705646b486177b59f9497da4d"} err="failed to get container status \"4ead25a0a8c3fae177b52dfcf7bae36d5641574705646b486177b59f9497da4d\": rpc error: code = NotFound desc = could not find container \"4ead25a0a8c3fae177b52dfcf7bae36d5641574705646b486177b59f9497da4d\": container with ID starting with 4ead25a0a8c3fae177b52dfcf7bae36d5641574705646b486177b59f9497da4d not found: ID does not exist"
Aug 04 18:41:32.709553 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.709505    2566 scope.go:117] "RemoveContainer" containerID="58953d7cd0610725f58da97c367b86229bb82ab222f207065480ca379fbc25f1"
Aug 04 18:41:32.709773 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:41:32.709756    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58953d7cd0610725f58da97c367b86229bb82ab222f207065480ca379fbc25f1\": container with ID starting with 58953d7cd0610725f58da97c367b86229bb82ab222f207065480ca379fbc25f1 not found: ID does not exist" containerID="58953d7cd0610725f58da97c367b86229bb82ab222f207065480ca379fbc25f1"
Aug 04 18:41:32.709820 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.709779    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58953d7cd0610725f58da97c367b86229bb82ab222f207065480ca379fbc25f1"} err="failed to get container status \"58953d7cd0610725f58da97c367b86229bb82ab222f207065480ca379fbc25f1\": rpc error: code = NotFound desc = could not find container \"58953d7cd0610725f58da97c367b86229bb82ab222f207065480ca379fbc25f1\": container with ID starting with 58953d7cd0610725f58da97c367b86229bb82ab222f207065480ca379fbc25f1 not found: ID does not exist"
Aug 04 18:41:32.709820 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.709798    2566 scope.go:117] "RemoveContainer" containerID="bcfaf9a7533eefcbbc214cc78218112ae8d40a2514240607c74c9678aa8817d6"
Aug 04 18:41:32.710016 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:41:32.709996    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcfaf9a7533eefcbbc214cc78218112ae8d40a2514240607c74c9678aa8817d6\": container with ID starting with bcfaf9a7533eefcbbc214cc78218112ae8d40a2514240607c74c9678aa8817d6 not found: ID does not exist" containerID="bcfaf9a7533eefcbbc214cc78218112ae8d40a2514240607c74c9678aa8817d6"
Aug 04 18:41:32.710073 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.710022    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcfaf9a7533eefcbbc214cc78218112ae8d40a2514240607c74c9678aa8817d6"} err="failed to get container status \"bcfaf9a7533eefcbbc214cc78218112ae8d40a2514240607c74c9678aa8817d6\": rpc error: code = NotFound desc = could not find container \"bcfaf9a7533eefcbbc214cc78218112ae8d40a2514240607c74c9678aa8817d6\": container with ID starting with bcfaf9a7533eefcbbc214cc78218112ae8d40a2514240607c74c9678aa8817d6 not found: ID does not exist"
Aug 04 18:41:32.715488 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:32.715466    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jsrtd"]
Aug 04 18:41:33.195158 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.195128    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" path="/var/lib/kubelet/pods/549308d8-e567-4ded-bba3-2adfd6a70dce/volumes"
Aug 04 18:41:33.487545 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487467    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"]
Aug 04 18:41:33.487815 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487803    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kube-rbac-proxy"
Aug 04 18:41:33.487861 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487816    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kube-rbac-proxy"
Aug 04 18:41:33.487861 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487830    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kserve-container"
Aug 04 18:41:33.487861 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487836    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kserve-container"
Aug 04 18:41:33.487861 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487848    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="storage-initializer"
Aug 04 18:41:33.487861 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487854    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="storage-initializer"
Aug 04 18:41:33.487861 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487861    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="kube-rbac-proxy"
Aug 04 18:41:33.488048 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487866    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="kube-rbac-proxy"
Aug 04 18:41:33.488048 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487878    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="kserve-container"
Aug 04 18:41:33.488048 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487882    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="kserve-container"
Aug 04 18:41:33.488048 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487895    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="storage-initializer"
Aug 04 18:41:33.488048 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487900    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="storage-initializer"
Aug 04 18:41:33.488048 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487950    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="kserve-container"
Aug 04 18:41:33.488048 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487957    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kube-rbac-proxy"
Aug 04 18:41:33.488048 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487965    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="302e1b89-32e3-4c00-bc7a-0523b4937770" containerName="kserve-container"
Aug 04 18:41:33.488048 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.487974    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="549308d8-e567-4ded-bba3-2adfd6a70dce" containerName="kube-rbac-proxy"
Aug 04 18:41:33.491142 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.491123    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.493726 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.493706    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Aug 04 18:41:33.493850 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.493820    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:41:33.493850 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.493842    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:41:33.493965 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.493859    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:41:33.493965 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.493923    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Aug 04 18:41:33.494623 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.494610    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:41:33.494660 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.494647    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:41:33.501977 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.501959    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"]
Aug 04 18:41:33.594420 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.594383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18617e83-2f92-4bf8-af23-fe3de8f340e6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.594599 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.594435    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18617e83-2f92-4bf8-af23-fe3de8f340e6-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.594599 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.594531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18617e83-2f92-4bf8-af23-fe3de8f340e6-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.594599 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.594577    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-457wn\" (UniqueName: \"kubernetes.io/projected/18617e83-2f92-4bf8-af23-fe3de8f340e6-kube-api-access-457wn\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.594716 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.594619    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18617e83-2f92-4bf8-af23-fe3de8f340e6-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.695088 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.695053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18617e83-2f92-4bf8-af23-fe3de8f340e6-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.695243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.695110    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18617e83-2f92-4bf8-af23-fe3de8f340e6-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.695243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.695141    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-457wn\" (UniqueName: \"kubernetes.io/projected/18617e83-2f92-4bf8-af23-fe3de8f340e6-kube-api-access-457wn\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.695243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.695166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18617e83-2f92-4bf8-af23-fe3de8f340e6-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.695243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.695218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18617e83-2f92-4bf8-af23-fe3de8f340e6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.695530 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.695508    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18617e83-2f92-4bf8-af23-fe3de8f340e6-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.695889 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.695865    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18617e83-2f92-4bf8-af23-fe3de8f340e6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.695982 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.695901    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18617e83-2f92-4bf8-af23-fe3de8f340e6-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.697569 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.697552    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18617e83-2f92-4bf8-af23-fe3de8f340e6-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.705170 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.705146    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-457wn\" (UniqueName: \"kubernetes.io/projected/18617e83-2f92-4bf8-af23-fe3de8f340e6-kube-api-access-457wn\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.802008 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.801973    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:33.943483 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:33.943459    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"]
Aug 04 18:41:33.945890 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:41:33.945859    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18617e83_2f92_4bf8_af23_fe3de8f340e6.slice/crio-355817a439d5d33c67fb6c4b01a7286426d03fd2ac4c4858ca20afbf91e431bd WatchSource:0}: Error finding container 355817a439d5d33c67fb6c4b01a7286426d03fd2ac4c4858ca20afbf91e431bd: Status 404 returned error can't find the container with id 355817a439d5d33c67fb6c4b01a7286426d03fd2ac4c4858ca20afbf91e431bd
Aug 04 18:41:34.695244 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:34.695206    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh" event={"ID":"18617e83-2f92-4bf8-af23-fe3de8f340e6","Type":"ContainerStarted","Data":"4082650dc8162b5e6b8d2ef4c9f1f4c9fc1e40b2069d72a500e5b76de14022cb"}
Aug 04 18:41:34.695667 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:34.695262    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh" event={"ID":"18617e83-2f92-4bf8-af23-fe3de8f340e6","Type":"ContainerStarted","Data":"355817a439d5d33c67fb6c4b01a7286426d03fd2ac4c4858ca20afbf91e431bd"}
Aug 04 18:41:38.716148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:38.716113    2566 generic.go:356] "Generic (PLEG): container finished" podID="18617e83-2f92-4bf8-af23-fe3de8f340e6" containerID="4082650dc8162b5e6b8d2ef4c9f1f4c9fc1e40b2069d72a500e5b76de14022cb" exitCode=0
Aug 04 18:41:38.716568 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:38.716185    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh" event={"ID":"18617e83-2f92-4bf8-af23-fe3de8f340e6","Type":"ContainerDied","Data":"4082650dc8162b5e6b8d2ef4c9f1f4c9fc1e40b2069d72a500e5b76de14022cb"}
Aug 04 18:41:39.722393 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:39.722339    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh" event={"ID":"18617e83-2f92-4bf8-af23-fe3de8f340e6","Type":"ContainerStarted","Data":"fa1a35f3654a9fb542ddc282100d831078edf7eb8e00235aec7b7296c222c699"}
Aug 04 18:41:39.722768 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:39.722423    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh" event={"ID":"18617e83-2f92-4bf8-af23-fe3de8f340e6","Type":"ContainerStarted","Data":"b8b8e7433a02b7278d1644dc333f4465ba7fa31b83af1eab29a1d03a13e4bc59"}
Aug 04 18:41:39.722768 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:39.722611    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:39.722768 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:39.722723    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:41:39.750651 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:39.750605    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh" podStartSLOduration=2.779118437 podStartE2EDuration="6.750589405s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.971470968s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:41:39.748150831 +0000 UTC m=+2499.038765003" watchObservedRunningTime="2026-08-04 18:41:39.750589405 +0000 UTC m=+2499.041203577"
Aug 04 18:41:45.732959 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:41:45.732916    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:42:15.746754 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:15.746722    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:42:23.571642 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:23.571609    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"]
Aug 04 18:42:23.572043 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:23.571834    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh" podUID="18617e83-2f92-4bf8-af23-fe3de8f340e6" containerName="kserve-container" containerID="cri-o://b8b8e7433a02b7278d1644dc333f4465ba7fa31b83af1eab29a1d03a13e4bc59" gracePeriod=30
Aug 04 18:42:23.572043 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:23.571879    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh" podUID="18617e83-2f92-4bf8-af23-fe3de8f340e6" containerName="kube-rbac-proxy" containerID="cri-o://fa1a35f3654a9fb542ddc282100d831078edf7eb8e00235aec7b7296c222c699" gracePeriod=30
Aug 04 18:42:23.934713 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:23.934623    2566 generic.go:356] "Generic (PLEG): container finished" podID="18617e83-2f92-4bf8-af23-fe3de8f340e6" containerID="fa1a35f3654a9fb542ddc282100d831078edf7eb8e00235aec7b7296c222c699" exitCode=2
Aug 04 18:42:23.934713 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:23.934672    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh" event={"ID":"18617e83-2f92-4bf8-af23-fe3de8f340e6","Type":"ContainerDied","Data":"fa1a35f3654a9fb542ddc282100d831078edf7eb8e00235aec7b7296c222c699"}
Aug 04 18:42:25.728150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:25.728102    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh" podUID="18617e83-2f92-4bf8-af23-fe3de8f340e6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Aug 04 18:42:28.740262 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:28.740217    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"]
Aug 04 18:42:28.743644 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:28.743624    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:28.746366 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:28.746323    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Aug 04 18:42:28.746521 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:28.746415    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Aug 04 18:42:28.756414 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:28.756388    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"]
Aug 04 18:42:28.899909 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:28.899807    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b998b40-c088-4e26-b7c9-a87214bf8be7-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-psscx\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:28.899909 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:28.899853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h75bm\" (UniqueName: \"kubernetes.io/projected/4b998b40-c088-4e26-b7c9-a87214bf8be7-kube-api-access-h75bm\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-psscx\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:28.899909 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:28.899885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b998b40-c088-4e26-b7c9-a87214bf8be7-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-psscx\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:28.900173 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:28.899944    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b998b40-c088-4e26-b7c9-a87214bf8be7-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-psscx\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:28.900173 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:28.900031    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b998b40-c088-4e26-b7c9-a87214bf8be7-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-psscx\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:29.000499 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:29.000470    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b998b40-c088-4e26-b7c9-a87214bf8be7-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-psscx\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:29.000693 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:29.000505    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h75bm\" (UniqueName: \"kubernetes.io/projected/4b998b40-c088-4e26-b7c9-a87214bf8be7-kube-api-access-h75bm\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-psscx\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:29.000693 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:29.000539    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b998b40-c088-4e26-b7c9-a87214bf8be7-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-psscx\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:29.000693 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:29.000563    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b998b40-c088-4e26-b7c9-a87214bf8be7-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-psscx\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:29.000693 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:29.000623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b998b40-c088-4e26-b7c9-a87214bf8be7-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-psscx\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:29.001039 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:29.001017    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b998b40-c088-4e26-b7c9-a87214bf8be7-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-psscx\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:29.001329 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:29.001310    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b998b40-c088-4e26-b7c9-a87214bf8be7-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-psscx\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:29.001388 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:29.001340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b998b40-c088-4e26-b7c9-a87214bf8be7-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-psscx\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:29.002932 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:29.002913    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b998b40-c088-4e26-b7c9-a87214bf8be7-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-psscx\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:29.011337 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:29.011307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h75bm\" (UniqueName: \"kubernetes.io/projected/4b998b40-c088-4e26-b7c9-a87214bf8be7-kube-api-access-h75bm\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-psscx\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:29.054273 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:29.054231    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:29.223418 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:29.223397    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"]
Aug 04 18:42:29.225376 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:42:29.225334    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b998b40_c088_4e26_b7c9_a87214bf8be7.slice/crio-b9d9c68e8a819373119f153e25aaa8b81079a0fe73e3d2a632db6d1320793908 WatchSource:0}: Error finding container b9d9c68e8a819373119f153e25aaa8b81079a0fe73e3d2a632db6d1320793908: Status 404 returned error can't find the container with id b9d9c68e8a819373119f153e25aaa8b81079a0fe73e3d2a632db6d1320793908
Aug 04 18:42:29.966050 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:29.966007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx" event={"ID":"4b998b40-c088-4e26-b7c9-a87214bf8be7","Type":"ContainerStarted","Data":"4cc60b66ee8b5675220f1e4854485874c05d05d02ce10f4e168fc1eaa2ed92b7"}
Aug 04 18:42:29.966445 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:29.966088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx" event={"ID":"4b998b40-c088-4e26-b7c9-a87214bf8be7","Type":"ContainerStarted","Data":"b9d9c68e8a819373119f153e25aaa8b81079a0fe73e3d2a632db6d1320793908"}
Aug 04 18:42:30.312958 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.312934    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:42:30.413820 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.413780    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18617e83-2f92-4bf8-af23-fe3de8f340e6-cabundle-cert\") pod \"18617e83-2f92-4bf8-af23-fe3de8f340e6\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") "
Aug 04 18:42:30.414022 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.413833    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18617e83-2f92-4bf8-af23-fe3de8f340e6-kube-rbac-proxy-sar-config\") pod \"18617e83-2f92-4bf8-af23-fe3de8f340e6\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") "
Aug 04 18:42:30.414022 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.413891    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-457wn\" (UniqueName: \"kubernetes.io/projected/18617e83-2f92-4bf8-af23-fe3de8f340e6-kube-api-access-457wn\") pod \"18617e83-2f92-4bf8-af23-fe3de8f340e6\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") "
Aug 04 18:42:30.414022 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.413937    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18617e83-2f92-4bf8-af23-fe3de8f340e6-proxy-tls\") pod \"18617e83-2f92-4bf8-af23-fe3de8f340e6\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") "
Aug 04 18:42:30.414022 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.413979    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18617e83-2f92-4bf8-af23-fe3de8f340e6-kserve-provision-location\") pod \"18617e83-2f92-4bf8-af23-fe3de8f340e6\" (UID: \"18617e83-2f92-4bf8-af23-fe3de8f340e6\") "
Aug 04 18:42:30.414255 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.414230    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18617e83-2f92-4bf8-af23-fe3de8f340e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "18617e83-2f92-4bf8-af23-fe3de8f340e6" (UID: "18617e83-2f92-4bf8-af23-fe3de8f340e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:42:30.414312 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.414292    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18617e83-2f92-4bf8-af23-fe3de8f340e6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "18617e83-2f92-4bf8-af23-fe3de8f340e6" (UID: "18617e83-2f92-4bf8-af23-fe3de8f340e6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:42:30.414375 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.414303    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18617e83-2f92-4bf8-af23-fe3de8f340e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "18617e83-2f92-4bf8-af23-fe3de8f340e6" (UID: "18617e83-2f92-4bf8-af23-fe3de8f340e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:42:30.416223 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.416199    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18617e83-2f92-4bf8-af23-fe3de8f340e6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "18617e83-2f92-4bf8-af23-fe3de8f340e6" (UID: "18617e83-2f92-4bf8-af23-fe3de8f340e6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:42:30.416333 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.416240    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18617e83-2f92-4bf8-af23-fe3de8f340e6-kube-api-access-457wn" (OuterVolumeSpecName: "kube-api-access-457wn") pod "18617e83-2f92-4bf8-af23-fe3de8f340e6" (UID: "18617e83-2f92-4bf8-af23-fe3de8f340e6"). InnerVolumeSpecName "kube-api-access-457wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:42:30.515115 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.515012    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18617e83-2f92-4bf8-af23-fe3de8f340e6-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:42:30.515115 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.515056    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18617e83-2f92-4bf8-af23-fe3de8f340e6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:42:30.515115 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.515068    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-457wn\" (UniqueName: \"kubernetes.io/projected/18617e83-2f92-4bf8-af23-fe3de8f340e6-kube-api-access-457wn\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:42:30.515115 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.515079    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18617e83-2f92-4bf8-af23-fe3de8f340e6-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:42:30.515115 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.515088    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18617e83-2f92-4bf8-af23-fe3de8f340e6-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:42:30.971906 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.971871    2566 generic.go:356] "Generic (PLEG): container finished" podID="18617e83-2f92-4bf8-af23-fe3de8f340e6" containerID="b8b8e7433a02b7278d1644dc333f4465ba7fa31b83af1eab29a1d03a13e4bc59" exitCode=0
Aug 04 18:42:30.972334 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.971939    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"
Aug 04 18:42:30.972334 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.971953    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh" event={"ID":"18617e83-2f92-4bf8-af23-fe3de8f340e6","Type":"ContainerDied","Data":"b8b8e7433a02b7278d1644dc333f4465ba7fa31b83af1eab29a1d03a13e4bc59"}
Aug 04 18:42:30.972334 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.972012    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh" event={"ID":"18617e83-2f92-4bf8-af23-fe3de8f340e6","Type":"ContainerDied","Data":"355817a439d5d33c67fb6c4b01a7286426d03fd2ac4c4858ca20afbf91e431bd"}
Aug 04 18:42:30.972334 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.972039    2566 scope.go:117] "RemoveContainer" containerID="fa1a35f3654a9fb542ddc282100d831078edf7eb8e00235aec7b7296c222c699"
Aug 04 18:42:30.981409 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.981390    2566 scope.go:117] "RemoveContainer" containerID="b8b8e7433a02b7278d1644dc333f4465ba7fa31b83af1eab29a1d03a13e4bc59"
Aug 04 18:42:30.989526 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.989508    2566 scope.go:117] "RemoveContainer" containerID="4082650dc8162b5e6b8d2ef4c9f1f4c9fc1e40b2069d72a500e5b76de14022cb"
Aug 04 18:42:30.998291 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.998161    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"]
Aug 04 18:42:30.998374 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.998314    2566 scope.go:117] "RemoveContainer" containerID="fa1a35f3654a9fb542ddc282100d831078edf7eb8e00235aec7b7296c222c699"
Aug 04 18:42:30.998615 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:42:30.998598    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa1a35f3654a9fb542ddc282100d831078edf7eb8e00235aec7b7296c222c699\": container with ID starting with fa1a35f3654a9fb542ddc282100d831078edf7eb8e00235aec7b7296c222c699 not found: ID does not exist" containerID="fa1a35f3654a9fb542ddc282100d831078edf7eb8e00235aec7b7296c222c699"
Aug 04 18:42:30.998670 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.998624    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1a35f3654a9fb542ddc282100d831078edf7eb8e00235aec7b7296c222c699"} err="failed to get container status \"fa1a35f3654a9fb542ddc282100d831078edf7eb8e00235aec7b7296c222c699\": rpc error: code = NotFound desc = could not find container \"fa1a35f3654a9fb542ddc282100d831078edf7eb8e00235aec7b7296c222c699\": container with ID starting with fa1a35f3654a9fb542ddc282100d831078edf7eb8e00235aec7b7296c222c699 not found: ID does not exist"
Aug 04 18:42:30.998670 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.998641    2566 scope.go:117] "RemoveContainer" containerID="b8b8e7433a02b7278d1644dc333f4465ba7fa31b83af1eab29a1d03a13e4bc59"
Aug 04 18:42:30.998894 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:42:30.998876    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b8e7433a02b7278d1644dc333f4465ba7fa31b83af1eab29a1d03a13e4bc59\": container with ID starting with b8b8e7433a02b7278d1644dc333f4465ba7fa31b83af1eab29a1d03a13e4bc59 not found: ID does not exist" containerID="b8b8e7433a02b7278d1644dc333f4465ba7fa31b83af1eab29a1d03a13e4bc59"
Aug 04 18:42:30.998944 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.998899    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b8e7433a02b7278d1644dc333f4465ba7fa31b83af1eab29a1d03a13e4bc59"} err="failed to get container status \"b8b8e7433a02b7278d1644dc333f4465ba7fa31b83af1eab29a1d03a13e4bc59\": rpc error: code = NotFound desc = could not find container \"b8b8e7433a02b7278d1644dc333f4465ba7fa31b83af1eab29a1d03a13e4bc59\": container with ID starting with b8b8e7433a02b7278d1644dc333f4465ba7fa31b83af1eab29a1d03a13e4bc59 not found: ID does not exist"
Aug 04 18:42:30.998944 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.998915    2566 scope.go:117] "RemoveContainer" containerID="4082650dc8162b5e6b8d2ef4c9f1f4c9fc1e40b2069d72a500e5b76de14022cb"
Aug 04 18:42:30.999143 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:42:30.999121    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4082650dc8162b5e6b8d2ef4c9f1f4c9fc1e40b2069d72a500e5b76de14022cb\": container with ID starting with 4082650dc8162b5e6b8d2ef4c9f1f4c9fc1e40b2069d72a500e5b76de14022cb not found: ID does not exist" containerID="4082650dc8162b5e6b8d2ef4c9f1f4c9fc1e40b2069d72a500e5b76de14022cb"
Aug 04 18:42:30.999202 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.999151    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4082650dc8162b5e6b8d2ef4c9f1f4c9fc1e40b2069d72a500e5b76de14022cb"} err="failed to get container status \"4082650dc8162b5e6b8d2ef4c9f1f4c9fc1e40b2069d72a500e5b76de14022cb\": rpc error: code = NotFound desc = could not find container \"4082650dc8162b5e6b8d2ef4c9f1f4c9fc1e40b2069d72a500e5b76de14022cb\": container with ID starting with 4082650dc8162b5e6b8d2ef4c9f1f4c9fc1e40b2069d72a500e5b76de14022cb not found: ID does not exist"
Aug 04 18:42:30.999920 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:30.999901    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-22drh"]
Aug 04 18:42:31.194439 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:31.194402    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18617e83-2f92-4bf8-af23-fe3de8f340e6" path="/var/lib/kubelet/pods/18617e83-2f92-4bf8-af23-fe3de8f340e6/volumes"
Aug 04 18:42:33.987344 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:33.987309    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b998b40-c088-4e26-b7c9-a87214bf8be7" containerID="4cc60b66ee8b5675220f1e4854485874c05d05d02ce10f4e168fc1eaa2ed92b7" exitCode=0
Aug 04 18:42:33.987774 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:33.987384    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx" event={"ID":"4b998b40-c088-4e26-b7c9-a87214bf8be7","Type":"ContainerDied","Data":"4cc60b66ee8b5675220f1e4854485874c05d05d02ce10f4e168fc1eaa2ed92b7"}
Aug 04 18:42:34.994254 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:34.994220    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx" event={"ID":"4b998b40-c088-4e26-b7c9-a87214bf8be7","Type":"ContainerStarted","Data":"0e42d568f8d9010f4f2bbf5146e79bd67ce22b4b3bc7766e31b9d545805dc7b5"}
Aug 04 18:42:34.994685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:34.994283    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx" event={"ID":"4b998b40-c088-4e26-b7c9-a87214bf8be7","Type":"ContainerStarted","Data":"95592a30f93566b8005f0c585c42744aafdf264b2699dcb64550fc7416aa9b63"}
Aug 04 18:42:34.994685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:34.994619    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:34.994685 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:34.994666    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:42:35.018286 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:35.018243    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx" podStartSLOduration=3.123679216 podStartE2EDuration="7.018228261s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.894549045s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:42:35.015782822 +0000 UTC m=+2554.306397019" watchObservedRunningTime="2026-08-04 18:42:35.018228261 +0000 UTC m=+2554.308842431"
Aug 04 18:42:41.003991 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:42:41.003958    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:43:11.009132 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:11.009103    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:43:18.815594 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:18.815562    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"]
Aug 04 18:43:18.816170 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:18.815840    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx" podUID="4b998b40-c088-4e26-b7c9-a87214bf8be7" containerName="kserve-container" containerID="cri-o://95592a30f93566b8005f0c585c42744aafdf264b2699dcb64550fc7416aa9b63" gracePeriod=30
Aug 04 18:43:18.816170 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:18.815920    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx" podUID="4b998b40-c088-4e26-b7c9-a87214bf8be7" containerName="kube-rbac-proxy" containerID="cri-o://0e42d568f8d9010f4f2bbf5146e79bd67ce22b4b3bc7766e31b9d545805dc7b5" gracePeriod=30
Aug 04 18:43:19.214863 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:19.214096    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b998b40-c088-4e26-b7c9-a87214bf8be7" containerID="0e42d568f8d9010f4f2bbf5146e79bd67ce22b4b3bc7766e31b9d545805dc7b5" exitCode=2
Aug 04 18:43:19.214863 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:19.214237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx" event={"ID":"4b998b40-c088-4e26-b7c9-a87214bf8be7","Type":"ContainerDied","Data":"0e42d568f8d9010f4f2bbf5146e79bd67ce22b4b3bc7766e31b9d545805dc7b5"}
Aug 04 18:43:21.000208 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:21.000168    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx" podUID="4b998b40-c088-4e26-b7c9-a87214bf8be7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Aug 04 18:43:25.461131 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:25.461104    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:43:25.505479 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:25.505382    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b998b40-c088-4e26-b7c9-a87214bf8be7-kube-rbac-proxy-sar-config\") pod \"4b998b40-c088-4e26-b7c9-a87214bf8be7\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") "
Aug 04 18:43:25.505479 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:25.505469    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b998b40-c088-4e26-b7c9-a87214bf8be7-cabundle-cert\") pod \"4b998b40-c088-4e26-b7c9-a87214bf8be7\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") "
Aug 04 18:43:25.505727 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:25.505518    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b998b40-c088-4e26-b7c9-a87214bf8be7-kserve-provision-location\") pod \"4b998b40-c088-4e26-b7c9-a87214bf8be7\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") "
Aug 04 18:43:25.505727 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:25.505540    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b998b40-c088-4e26-b7c9-a87214bf8be7-proxy-tls\") pod \"4b998b40-c088-4e26-b7c9-a87214bf8be7\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") "
Aug 04 18:43:25.505727 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:25.505562    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h75bm\" (UniqueName: \"kubernetes.io/projected/4b998b40-c088-4e26-b7c9-a87214bf8be7-kube-api-access-h75bm\") pod \"4b998b40-c088-4e26-b7c9-a87214bf8be7\" (UID: \"4b998b40-c088-4e26-b7c9-a87214bf8be7\") "
Aug 04 18:43:25.505887 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:25.505787    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b998b40-c088-4e26-b7c9-a87214bf8be7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4b998b40-c088-4e26-b7c9-a87214bf8be7" (UID: "4b998b40-c088-4e26-b7c9-a87214bf8be7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:43:25.505887 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:25.505844    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b998b40-c088-4e26-b7c9-a87214bf8be7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4b998b40-c088-4e26-b7c9-a87214bf8be7" (UID: "4b998b40-c088-4e26-b7c9-a87214bf8be7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:43:25.505887 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:25.505860    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b998b40-c088-4e26-b7c9-a87214bf8be7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b998b40-c088-4e26-b7c9-a87214bf8be7" (UID: "4b998b40-c088-4e26-b7c9-a87214bf8be7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:43:25.507724 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:25.507703    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b998b40-c088-4e26-b7c9-a87214bf8be7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b998b40-c088-4e26-b7c9-a87214bf8be7" (UID: "4b998b40-c088-4e26-b7c9-a87214bf8be7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:43:25.507815 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:25.507773    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b998b40-c088-4e26-b7c9-a87214bf8be7-kube-api-access-h75bm" (OuterVolumeSpecName: "kube-api-access-h75bm") pod "4b998b40-c088-4e26-b7c9-a87214bf8be7" (UID: "4b998b40-c088-4e26-b7c9-a87214bf8be7"). InnerVolumeSpecName "kube-api-access-h75bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:43:25.606686 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:25.606647    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b998b40-c088-4e26-b7c9-a87214bf8be7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:43:25.606686 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:25.606677    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b998b40-c088-4e26-b7c9-a87214bf8be7-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:43:25.606686 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:25.606688    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b998b40-c088-4e26-b7c9-a87214bf8be7-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:43:25.606686 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:25.606699    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b998b40-c088-4e26-b7c9-a87214bf8be7-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:43:25.606960 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:25.606708    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h75bm\" (UniqueName: \"kubernetes.io/projected/4b998b40-c088-4e26-b7c9-a87214bf8be7-kube-api-access-h75bm\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:43:26.248904 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:26.248871    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b998b40-c088-4e26-b7c9-a87214bf8be7" containerID="95592a30f93566b8005f0c585c42744aafdf264b2699dcb64550fc7416aa9b63" exitCode=0
Aug 04 18:43:26.249073 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:26.248923    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"
Aug 04 18:43:26.249073 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:26.248945    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx" event={"ID":"4b998b40-c088-4e26-b7c9-a87214bf8be7","Type":"ContainerDied","Data":"95592a30f93566b8005f0c585c42744aafdf264b2699dcb64550fc7416aa9b63"}
Aug 04 18:43:26.249073 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:26.249006    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx" event={"ID":"4b998b40-c088-4e26-b7c9-a87214bf8be7","Type":"ContainerDied","Data":"b9d9c68e8a819373119f153e25aaa8b81079a0fe73e3d2a632db6d1320793908"}
Aug 04 18:43:26.249073 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:26.249034    2566 scope.go:117] "RemoveContainer" containerID="0e42d568f8d9010f4f2bbf5146e79bd67ce22b4b3bc7766e31b9d545805dc7b5"
Aug 04 18:43:26.257744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:26.257726    2566 scope.go:117] "RemoveContainer" containerID="95592a30f93566b8005f0c585c42744aafdf264b2699dcb64550fc7416aa9b63"
Aug 04 18:43:26.265046 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:26.265027    2566 scope.go:117] "RemoveContainer" containerID="4cc60b66ee8b5675220f1e4854485874c05d05d02ce10f4e168fc1eaa2ed92b7"
Aug 04 18:43:26.272642 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:26.272616    2566 scope.go:117] "RemoveContainer" containerID="0e42d568f8d9010f4f2bbf5146e79bd67ce22b4b3bc7766e31b9d545805dc7b5"
Aug 04 18:43:26.272942 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:43:26.272920    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e42d568f8d9010f4f2bbf5146e79bd67ce22b4b3bc7766e31b9d545805dc7b5\": container with ID starting with 0e42d568f8d9010f4f2bbf5146e79bd67ce22b4b3bc7766e31b9d545805dc7b5 not found: ID does not exist" containerID="0e42d568f8d9010f4f2bbf5146e79bd67ce22b4b3bc7766e31b9d545805dc7b5"
Aug 04 18:43:26.273071 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:26.272957    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e42d568f8d9010f4f2bbf5146e79bd67ce22b4b3bc7766e31b9d545805dc7b5"} err="failed to get container status \"0e42d568f8d9010f4f2bbf5146e79bd67ce22b4b3bc7766e31b9d545805dc7b5\": rpc error: code = NotFound desc = could not find container \"0e42d568f8d9010f4f2bbf5146e79bd67ce22b4b3bc7766e31b9d545805dc7b5\": container with ID starting with 0e42d568f8d9010f4f2bbf5146e79bd67ce22b4b3bc7766e31b9d545805dc7b5 not found: ID does not exist"
Aug 04 18:43:26.273071 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:26.272975    2566 scope.go:117] "RemoveContainer" containerID="95592a30f93566b8005f0c585c42744aafdf264b2699dcb64550fc7416aa9b63"
Aug 04 18:43:26.273477 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:43:26.273423    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95592a30f93566b8005f0c585c42744aafdf264b2699dcb64550fc7416aa9b63\": container with ID starting with 95592a30f93566b8005f0c585c42744aafdf264b2699dcb64550fc7416aa9b63 not found: ID does not exist" containerID="95592a30f93566b8005f0c585c42744aafdf264b2699dcb64550fc7416aa9b63"
Aug 04 18:43:26.273477 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:26.273460    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95592a30f93566b8005f0c585c42744aafdf264b2699dcb64550fc7416aa9b63"} err="failed to get container status \"95592a30f93566b8005f0c585c42744aafdf264b2699dcb64550fc7416aa9b63\": rpc error: code = NotFound desc = could not find container \"95592a30f93566b8005f0c585c42744aafdf264b2699dcb64550fc7416aa9b63\": container with ID starting with 95592a30f93566b8005f0c585c42744aafdf264b2699dcb64550fc7416aa9b63 not found: ID does not exist"
Aug 04 18:43:26.273665 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:26.273482    2566 scope.go:117] "RemoveContainer" containerID="4cc60b66ee8b5675220f1e4854485874c05d05d02ce10f4e168fc1eaa2ed92b7"
Aug 04 18:43:26.273879 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:43:26.273852    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc60b66ee8b5675220f1e4854485874c05d05d02ce10f4e168fc1eaa2ed92b7\": container with ID starting with 4cc60b66ee8b5675220f1e4854485874c05d05d02ce10f4e168fc1eaa2ed92b7 not found: ID does not exist" containerID="4cc60b66ee8b5675220f1e4854485874c05d05d02ce10f4e168fc1eaa2ed92b7"
Aug 04 18:43:26.273982 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:26.273915    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc60b66ee8b5675220f1e4854485874c05d05d02ce10f4e168fc1eaa2ed92b7"} err="failed to get container status \"4cc60b66ee8b5675220f1e4854485874c05d05d02ce10f4e168fc1eaa2ed92b7\": rpc error: code = NotFound desc = could not find container \"4cc60b66ee8b5675220f1e4854485874c05d05d02ce10f4e168fc1eaa2ed92b7\": container with ID starting with 4cc60b66ee8b5675220f1e4854485874c05d05d02ce10f4e168fc1eaa2ed92b7 not found: ID does not exist"
Aug 04 18:43:26.275307 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:26.275290    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"]
Aug 04 18:43:26.280214 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:26.280193    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-psscx"]
Aug 04 18:43:27.194005 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:43:27.193975    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b998b40-c088-4e26-b7c9-a87214bf8be7" path="/var/lib/kubelet/pods/4b998b40-c088-4e26-b7c9-a87214bf8be7/volumes"
Aug 04 18:44:33.798208 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798172    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"]
Aug 04 18:44:33.798748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798603    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18617e83-2f92-4bf8-af23-fe3de8f340e6" containerName="storage-initializer"
Aug 04 18:44:33.798748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798615    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="18617e83-2f92-4bf8-af23-fe3de8f340e6" containerName="storage-initializer"
Aug 04 18:44:33.798748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798624    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18617e83-2f92-4bf8-af23-fe3de8f340e6" containerName="kserve-container"
Aug 04 18:44:33.798748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798629    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="18617e83-2f92-4bf8-af23-fe3de8f340e6" containerName="kserve-container"
Aug 04 18:44:33.798748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798644    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b998b40-c088-4e26-b7c9-a87214bf8be7" containerName="storage-initializer"
Aug 04 18:44:33.798748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798650    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b998b40-c088-4e26-b7c9-a87214bf8be7" containerName="storage-initializer"
Aug 04 18:44:33.798748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798657    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b998b40-c088-4e26-b7c9-a87214bf8be7" containerName="kserve-container"
Aug 04 18:44:33.798748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798663    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b998b40-c088-4e26-b7c9-a87214bf8be7" containerName="kserve-container"
Aug 04 18:44:33.798748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798673    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b998b40-c088-4e26-b7c9-a87214bf8be7" containerName="kube-rbac-proxy"
Aug 04 18:44:33.798748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798679    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b998b40-c088-4e26-b7c9-a87214bf8be7" containerName="kube-rbac-proxy"
Aug 04 18:44:33.798748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798694    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18617e83-2f92-4bf8-af23-fe3de8f340e6" containerName="kube-rbac-proxy"
Aug 04 18:44:33.798748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798699    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="18617e83-2f92-4bf8-af23-fe3de8f340e6" containerName="kube-rbac-proxy"
Aug 04 18:44:33.798748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798751    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="18617e83-2f92-4bf8-af23-fe3de8f340e6" containerName="kserve-container"
Aug 04 18:44:33.799184 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798758    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b998b40-c088-4e26-b7c9-a87214bf8be7" containerName="kserve-container"
Aug 04 18:44:33.799184 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798768    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b998b40-c088-4e26-b7c9-a87214bf8be7" containerName="kube-rbac-proxy"
Aug 04 18:44:33.799184 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.798775    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="18617e83-2f92-4bf8-af23-fe3de8f340e6" containerName="kube-rbac-proxy"
Aug 04 18:44:33.801964 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.801947    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:33.804862 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.804833    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Aug 04 18:44:33.805006 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.804941    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:44:33.805082 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.805034    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:44:33.805082 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.805047    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Aug 04 18:44:33.805317 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.805297    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:44:33.805776 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.805760    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:44:33.805863 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.805846    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:44:33.820013 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.819990    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"]
Aug 04 18:44:33.920714 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.920677    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70ab8fde-a54a-4501-96f2-5481bbad76cc-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:33.920714 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.920717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftnpx\" (UniqueName: \"kubernetes.io/projected/70ab8fde-a54a-4501-96f2-5481bbad76cc-kube-api-access-ftnpx\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:33.920970 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.920746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70ab8fde-a54a-4501-96f2-5481bbad76cc-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:33.920970 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.920816    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70ab8fde-a54a-4501-96f2-5481bbad76cc-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:33.920970 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:33.920875    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70ab8fde-a54a-4501-96f2-5481bbad76cc-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:34.022059 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:34.022010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70ab8fde-a54a-4501-96f2-5481bbad76cc-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:34.022286 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:34.022095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70ab8fde-a54a-4501-96f2-5481bbad76cc-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:34.022286 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:34.022123    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftnpx\" (UniqueName: \"kubernetes.io/projected/70ab8fde-a54a-4501-96f2-5481bbad76cc-kube-api-access-ftnpx\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:34.022286 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:34.022152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70ab8fde-a54a-4501-96f2-5481bbad76cc-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:34.022286 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:34.022177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70ab8fde-a54a-4501-96f2-5481bbad76cc-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:34.022654 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:34.022630    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70ab8fde-a54a-4501-96f2-5481bbad76cc-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:34.022809 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:34.022789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70ab8fde-a54a-4501-96f2-5481bbad76cc-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:34.022910 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:34.022889    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70ab8fde-a54a-4501-96f2-5481bbad76cc-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:34.024793 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:34.024774    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70ab8fde-a54a-4501-96f2-5481bbad76cc-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:34.032875 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:34.032845    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftnpx\" (UniqueName: \"kubernetes.io/projected/70ab8fde-a54a-4501-96f2-5481bbad76cc-kube-api-access-ftnpx\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:34.111946 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:34.111917    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:34.246505 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:34.246461    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"]
Aug 04 18:44:34.248835 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:44:34.248809    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ab8fde_a54a_4501_96f2_5481bbad76cc.slice/crio-5356870de1d886eeba301877a400b9c7137352ff3820a4c4c258d403137ba197 WatchSource:0}: Error finding container 5356870de1d886eeba301877a400b9c7137352ff3820a4c4c258d403137ba197: Status 404 returned error can't find the container with id 5356870de1d886eeba301877a400b9c7137352ff3820a4c4c258d403137ba197
Aug 04 18:44:34.547040 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:34.546956    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6" event={"ID":"70ab8fde-a54a-4501-96f2-5481bbad76cc","Type":"ContainerStarted","Data":"78b4beaaeec1d58e8659f3c5aeb34cc817fb32a37dd364568b9f3fbba71c7e9a"}
Aug 04 18:44:34.547040 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:34.547027    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6" event={"ID":"70ab8fde-a54a-4501-96f2-5481bbad76cc","Type":"ContainerStarted","Data":"5356870de1d886eeba301877a400b9c7137352ff3820a4c4c258d403137ba197"}
Aug 04 18:44:38.565845 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:38.565811    2566 generic.go:356] "Generic (PLEG): container finished" podID="70ab8fde-a54a-4501-96f2-5481bbad76cc" containerID="78b4beaaeec1d58e8659f3c5aeb34cc817fb32a37dd364568b9f3fbba71c7e9a" exitCode=0
Aug 04 18:44:38.566213 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:38.565884    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6" event={"ID":"70ab8fde-a54a-4501-96f2-5481bbad76cc","Type":"ContainerDied","Data":"78b4beaaeec1d58e8659f3c5aeb34cc817fb32a37dd364568b9f3fbba71c7e9a"}
Aug 04 18:44:39.573000 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:39.572955    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6" event={"ID":"70ab8fde-a54a-4501-96f2-5481bbad76cc","Type":"ContainerStarted","Data":"796f4acdd773d13036a48a6558c9fa1f634d14d1b4a76eccb5b57f3c5b7cab33"}
Aug 04 18:44:39.573398 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:39.573040    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6" event={"ID":"70ab8fde-a54a-4501-96f2-5481bbad76cc","Type":"ContainerStarted","Data":"b39f53e8bf88be9aa639bc3305feb0982b456a7ee4fb36a772601c714afead5c"}
Aug 04 18:44:39.573398 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:39.573262    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:39.597452 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:39.597401    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6" podStartSLOduration=2.809320357 podStartE2EDuration="6.597385242s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.788064885s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:44:39.595751594 +0000 UTC m=+2678.886365787" watchObservedRunningTime="2026-08-04 18:44:39.597385242 +0000 UTC m=+2678.887999416"
Aug 04 18:44:40.578396 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:40.578365    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:44:46.589748 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:44:46.589719    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:45:09.253146 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.253113    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"]
Aug 04 18:45:09.257028 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.257008    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.259484 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.259463    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Aug 04 18:45:09.259594 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.259500    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Aug 04 18:45:09.269703 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.269672    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"]
Aug 04 18:45:09.331573 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.331535    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.331790 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.331602    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pb6f\" (UniqueName: \"kubernetes.io/projected/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kube-api-access-6pb6f\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.331790 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.331633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78e7bc71-2810-49dd-83c8-2cc9997d28b1-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.331790 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.331669    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78e7bc71-2810-49dd-83c8-2cc9997d28b1-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.331790 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.331743    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.432298 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.432256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.432509 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.432326    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pb6f\" (UniqueName: \"kubernetes.io/projected/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kube-api-access-6pb6f\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.432509 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.432387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78e7bc71-2810-49dd-83c8-2cc9997d28b1-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.432509 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.432426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78e7bc71-2810-49dd-83c8-2cc9997d28b1-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.432509 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.432470    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.432872 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.432849    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.433010 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.432991    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.433115 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.433093    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78e7bc71-2810-49dd-83c8-2cc9997d28b1-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.434923 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.434903    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78e7bc71-2810-49dd-83c8-2cc9997d28b1-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.442253 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.442228    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pb6f\" (UniqueName: \"kubernetes.io/projected/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kube-api-access-6pb6f\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.567500 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.567468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:09.708270 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:09.708240    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"]
Aug 04 18:45:09.711834 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:45:09.711812    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e7bc71_2810_49dd_83c8_2cc9997d28b1.slice/crio-37fbc4b37d367e426411c0ed3eb03546061963cc42b8bb31a66db8105363d909 WatchSource:0}: Error finding container 37fbc4b37d367e426411c0ed3eb03546061963cc42b8bb31a66db8105363d909: Status 404 returned error can't find the container with id 37fbc4b37d367e426411c0ed3eb03546061963cc42b8bb31a66db8105363d909
Aug 04 18:45:10.714875 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:10.714838    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv" event={"ID":"78e7bc71-2810-49dd-83c8-2cc9997d28b1","Type":"ContainerStarted","Data":"14f7af37356bf19b6dc9683b9f154eadc0e12dcdd132a6981793dde745d06608"}
Aug 04 18:45:10.715323 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:10.714900    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv" event={"ID":"78e7bc71-2810-49dd-83c8-2cc9997d28b1","Type":"ContainerStarted","Data":"37fbc4b37d367e426411c0ed3eb03546061963cc42b8bb31a66db8105363d909"}
Aug 04 18:45:13.730825 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:13.730789    2566 generic.go:356] "Generic (PLEG): container finished" podID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerID="14f7af37356bf19b6dc9683b9f154eadc0e12dcdd132a6981793dde745d06608" exitCode=0
Aug 04 18:45:13.731310 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:13.730870    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv" event={"ID":"78e7bc71-2810-49dd-83c8-2cc9997d28b1","Type":"ContainerDied","Data":"14f7af37356bf19b6dc9683b9f154eadc0e12dcdd132a6981793dde745d06608"}
Aug 04 18:45:14.737644 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:14.737606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv" event={"ID":"78e7bc71-2810-49dd-83c8-2cc9997d28b1","Type":"ContainerStarted","Data":"f6a8df18a4123c604c20ea8e342cc872f275093346547526364b1469a08e3c15"}
Aug 04 18:45:14.738058 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:14.737698    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv" event={"ID":"78e7bc71-2810-49dd-83c8-2cc9997d28b1","Type":"ContainerStarted","Data":"3899c86ec61131f8e184adcff0f458d557813a91ab3cecefd87fc83b844aebc1"}
Aug 04 18:45:14.738058 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:14.737882    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:14.765265 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:14.765214    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv" podStartSLOduration=2.00176187 podStartE2EDuration="5.765197821s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.763435951s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:45:14.763992181 +0000 UTC m=+2714.054606363" watchObservedRunningTime="2026-08-04 18:45:14.765197821 +0000 UTC m=+2714.055811991"
Aug 04 18:45:15.743032 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:15.743001    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:16.593499 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:16.593471    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:45:21.753347 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:21.753320    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:45:23.821589 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:23.821556    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"]
Aug 04 18:45:23.821980 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:23.821767    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6" podUID="70ab8fde-a54a-4501-96f2-5481bbad76cc" containerName="kserve-container" containerID="cri-o://b39f53e8bf88be9aa639bc3305feb0982b456a7ee4fb36a772601c714afead5c" gracePeriod=30
Aug 04 18:45:23.821980 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:23.821814    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6" podUID="70ab8fde-a54a-4501-96f2-5481bbad76cc" containerName="kube-rbac-proxy" containerID="cri-o://796f4acdd773d13036a48a6558c9fa1f634d14d1b4a76eccb5b57f3c5b7cab33" gracePeriod=30
Aug 04 18:45:24.786999 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:24.786970    2566 generic.go:356] "Generic (PLEG): container finished" podID="70ab8fde-a54a-4501-96f2-5481bbad76cc" containerID="796f4acdd773d13036a48a6558c9fa1f634d14d1b4a76eccb5b57f3c5b7cab33" exitCode=2
Aug 04 18:45:24.786999 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:24.786993    2566 generic.go:356] "Generic (PLEG): container finished" podID="70ab8fde-a54a-4501-96f2-5481bbad76cc" containerID="b39f53e8bf88be9aa639bc3305feb0982b456a7ee4fb36a772601c714afead5c" exitCode=0
Aug 04 18:45:24.787231 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:24.787043    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6" event={"ID":"70ab8fde-a54a-4501-96f2-5481bbad76cc","Type":"ContainerDied","Data":"796f4acdd773d13036a48a6558c9fa1f634d14d1b4a76eccb5b57f3c5b7cab33"}
Aug 04 18:45:24.787231 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:24.787096    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6" event={"ID":"70ab8fde-a54a-4501-96f2-5481bbad76cc","Type":"ContainerDied","Data":"b39f53e8bf88be9aa639bc3305feb0982b456a7ee4fb36a772601c714afead5c"}
Aug 04 18:45:24.875048 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:24.875024    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:45:24.974689 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:24.974604    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70ab8fde-a54a-4501-96f2-5481bbad76cc-proxy-tls\") pod \"70ab8fde-a54a-4501-96f2-5481bbad76cc\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") "
Aug 04 18:45:24.974689 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:24.974658    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70ab8fde-a54a-4501-96f2-5481bbad76cc-cabundle-cert\") pod \"70ab8fde-a54a-4501-96f2-5481bbad76cc\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") "
Aug 04 18:45:24.974689 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:24.974679    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftnpx\" (UniqueName: \"kubernetes.io/projected/70ab8fde-a54a-4501-96f2-5481bbad76cc-kube-api-access-ftnpx\") pod \"70ab8fde-a54a-4501-96f2-5481bbad76cc\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") "
Aug 04 18:45:24.974949 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:24.974773    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70ab8fde-a54a-4501-96f2-5481bbad76cc-kube-rbac-proxy-sar-config\") pod \"70ab8fde-a54a-4501-96f2-5481bbad76cc\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") "
Aug 04 18:45:24.974949 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:24.974821    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70ab8fde-a54a-4501-96f2-5481bbad76cc-kserve-provision-location\") pod \"70ab8fde-a54a-4501-96f2-5481bbad76cc\" (UID: \"70ab8fde-a54a-4501-96f2-5481bbad76cc\") "
Aug 04 18:45:24.975125 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:24.975098    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70ab8fde-a54a-4501-96f2-5481bbad76cc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "70ab8fde-a54a-4501-96f2-5481bbad76cc" (UID: "70ab8fde-a54a-4501-96f2-5481bbad76cc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:45:24.975209 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:24.975137    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70ab8fde-a54a-4501-96f2-5481bbad76cc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "70ab8fde-a54a-4501-96f2-5481bbad76cc" (UID: "70ab8fde-a54a-4501-96f2-5481bbad76cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:45:24.975267 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:24.975219    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ab8fde-a54a-4501-96f2-5481bbad76cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70ab8fde-a54a-4501-96f2-5481bbad76cc" (UID: "70ab8fde-a54a-4501-96f2-5481bbad76cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:45:24.976775 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:24.976748    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ab8fde-a54a-4501-96f2-5481bbad76cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70ab8fde-a54a-4501-96f2-5481bbad76cc" (UID: "70ab8fde-a54a-4501-96f2-5481bbad76cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:45:24.976870 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:24.976755    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ab8fde-a54a-4501-96f2-5481bbad76cc-kube-api-access-ftnpx" (OuterVolumeSpecName: "kube-api-access-ftnpx") pod "70ab8fde-a54a-4501-96f2-5481bbad76cc" (UID: "70ab8fde-a54a-4501-96f2-5481bbad76cc"). InnerVolumeSpecName "kube-api-access-ftnpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:45:25.075881 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:25.075843    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70ab8fde-a54a-4501-96f2-5481bbad76cc-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:45:25.075881 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:25.075873    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftnpx\" (UniqueName: \"kubernetes.io/projected/70ab8fde-a54a-4501-96f2-5481bbad76cc-kube-api-access-ftnpx\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:45:25.075881 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:25.075885    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70ab8fde-a54a-4501-96f2-5481bbad76cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:45:25.076134 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:25.075896    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70ab8fde-a54a-4501-96f2-5481bbad76cc-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:45:25.076134 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:25.075908    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70ab8fde-a54a-4501-96f2-5481bbad76cc-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:45:25.793864 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:25.793839    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"
Aug 04 18:45:25.794029 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:25.793870    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6" event={"ID":"70ab8fde-a54a-4501-96f2-5481bbad76cc","Type":"ContainerDied","Data":"5356870de1d886eeba301877a400b9c7137352ff3820a4c4c258d403137ba197"}
Aug 04 18:45:25.794029 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:25.793934    2566 scope.go:117] "RemoveContainer" containerID="796f4acdd773d13036a48a6558c9fa1f634d14d1b4a76eccb5b57f3c5b7cab33"
Aug 04 18:45:25.802166 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:25.802148    2566 scope.go:117] "RemoveContainer" containerID="b39f53e8bf88be9aa639bc3305feb0982b456a7ee4fb36a772601c714afead5c"
Aug 04 18:45:25.809661 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:25.809645    2566 scope.go:117] "RemoveContainer" containerID="78b4beaaeec1d58e8659f3c5aeb34cc817fb32a37dd364568b9f3fbba71c7e9a"
Aug 04 18:45:25.819590 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:25.819561    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"]
Aug 04 18:45:25.821479 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:25.821459    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdds6"]
Aug 04 18:45:27.194632 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:27.194599    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70ab8fde-a54a-4501-96f2-5481bbad76cc" path="/var/lib/kubelet/pods/70ab8fde-a54a-4501-96f2-5481bbad76cc/volumes"
Aug 04 18:45:51.846230 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:51.846178    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv" podUID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Aug 04 18:45:54.260558 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.260523    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"]
Aug 04 18:45:54.261005 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.260919    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70ab8fde-a54a-4501-96f2-5481bbad76cc" containerName="kube-rbac-proxy"
Aug 04 18:45:54.261005 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.260931    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="70ab8fde-a54a-4501-96f2-5481bbad76cc" containerName="kube-rbac-proxy"
Aug 04 18:45:54.261005 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.260942    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70ab8fde-a54a-4501-96f2-5481bbad76cc" containerName="storage-initializer"
Aug 04 18:45:54.261005 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.260947    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="70ab8fde-a54a-4501-96f2-5481bbad76cc" containerName="storage-initializer"
Aug 04 18:45:54.261005 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.260961    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70ab8fde-a54a-4501-96f2-5481bbad76cc" containerName="kserve-container"
Aug 04 18:45:54.261005 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.260967    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="70ab8fde-a54a-4501-96f2-5481bbad76cc" containerName="kserve-container"
Aug 04 18:45:54.261202 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.261039    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="70ab8fde-a54a-4501-96f2-5481bbad76cc" containerName="kserve-container"
Aug 04 18:45:54.261202 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.261048    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="70ab8fde-a54a-4501-96f2-5481bbad76cc" containerName="kube-rbac-proxy"
Aug 04 18:45:54.264373 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.264339    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:54.267368 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.267326    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Aug 04 18:45:54.267698 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.267677    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Aug 04 18:45:54.276811 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.276788    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"]
Aug 04 18:45:54.336222 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.336184    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:54.336470 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.336246    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:54.336470 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.336317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c72qb\" (UniqueName: \"kubernetes.io/projected/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kube-api-access-c72qb\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:54.336470 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.336455    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:54.336613 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.336500    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:54.437512 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.437467    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:54.437512 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.437514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c72qb\" (UniqueName: \"kubernetes.io/projected/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kube-api-access-c72qb\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:54.437784 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.437588    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:54.437784 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.437630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:54.437784 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.437665    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:54.437947 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:45:54.437785    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Aug 04 18:45:54.437947 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:45:54.437890    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-proxy-tls podName:7f36a27d-4808-4f14-b2ff-5c7f6c0b3165 nodeName:}" failed. No retries permitted until 2026-08-04 18:45:54.937868478 +0000 UTC m=+2754.228482627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr" (UID: "7f36a27d-4808-4f14-b2ff-5c7f6c0b3165") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Aug 04 18:45:54.437947 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.437908    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:54.438277 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.438256    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:54.438395 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.438335    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:54.449142 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.449115    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c72qb\" (UniqueName: \"kubernetes.io/projected/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kube-api-access-c72qb\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:54.942187 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.942153    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:54.944568 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:54.944545    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:55.175014 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:55.174976    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:45:55.310229 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:55.310198    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"]
Aug 04 18:45:55.313827 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:45:55.313802    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f36a27d_4808_4f14_b2ff_5c7f6c0b3165.slice/crio-15f2f134c945df9c32b2f8f9069339e61fe855048c27c2316af0164ac1b8afaa WatchSource:0}: Error finding container 15f2f134c945df9c32b2f8f9069339e61fe855048c27c2316af0164ac1b8afaa: Status 404 returned error can't find the container with id 15f2f134c945df9c32b2f8f9069339e61fe855048c27c2316af0164ac1b8afaa
Aug 04 18:45:55.929188 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:55.929151    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr" event={"ID":"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165","Type":"ContainerStarted","Data":"a8d894a13c6997b49e2f39694caabbb6c8c60e6f0bb1eeede3b769bcbb96d47e"}
Aug 04 18:45:55.929384 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:55.929222    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr" event={"ID":"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165","Type":"ContainerStarted","Data":"15f2f134c945df9c32b2f8f9069339e61fe855048c27c2316af0164ac1b8afaa"}
Aug 04 18:45:59.948431 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:59.948393    2566 generic.go:356] "Generic (PLEG): container finished" podID="7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" containerID="a8d894a13c6997b49e2f39694caabbb6c8c60e6f0bb1eeede3b769bcbb96d47e" exitCode=0
Aug 04 18:45:59.948812 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:45:59.948456    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr" event={"ID":"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165","Type":"ContainerDied","Data":"a8d894a13c6997b49e2f39694caabbb6c8c60e6f0bb1eeede3b769bcbb96d47e"}
Aug 04 18:46:00.955834 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:00.955800    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr" event={"ID":"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165","Type":"ContainerStarted","Data":"3c84e558c26ffcd02265a2e93671e46c5523fa13bf7faebf574240cfbe5166c8"}
Aug 04 18:46:00.956219 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:00.955866    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr" event={"ID":"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165","Type":"ContainerStarted","Data":"8a4443fd98ea2e2fa0a6176d7d43449afa285349a8b47bcacc686d2d4a2a2e71"}
Aug 04 18:46:00.956219 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:00.956067    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:46:00.984210 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:00.984149    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr" podStartSLOduration=3.079030758 podStartE2EDuration="6.98413129s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.905100532s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:46:00.983553267 +0000 UTC m=+2760.274167438" watchObservedRunningTime="2026-08-04 18:46:00.98413129 +0000 UTC m=+2760.274745460"
Aug 04 18:46:01.756755 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:01.756721    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:46:01.960680 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:01.960650    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:46:07.971961 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:07.971935    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:46:09.336089 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:09.336054    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"]
Aug 04 18:46:09.336536 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:09.336298    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv" podUID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerName="kserve-container" containerID="cri-o://3899c86ec61131f8e184adcff0f458d557813a91ab3cecefd87fc83b844aebc1" gracePeriod=30
Aug 04 18:46:09.336536 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:09.336348    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv" podUID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerName="kube-rbac-proxy" containerID="cri-o://f6a8df18a4123c604c20ea8e342cc872f275093346547526364b1469a08e3c15" gracePeriod=30
Aug 04 18:46:10.003720 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:10.003690    2566 generic.go:356] "Generic (PLEG): container finished" podID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerID="f6a8df18a4123c604c20ea8e342cc872f275093346547526364b1469a08e3c15" exitCode=2
Aug 04 18:46:10.003899 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:10.003765    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv" event={"ID":"78e7bc71-2810-49dd-83c8-2cc9997d28b1","Type":"ContainerDied","Data":"f6a8df18a4123c604c20ea8e342cc872f275093346547526364b1469a08e3c15"}
Aug 04 18:46:11.749217 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:11.749172    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv" podUID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Aug 04 18:46:12.795588 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:12.795542    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv" podUID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.65:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Aug 04 18:46:16.749289 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:16.749241    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv" podUID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Aug 04 18:46:17.038308 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.038280    2566 generic.go:356] "Generic (PLEG): container finished" podID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerID="3899c86ec61131f8e184adcff0f458d557813a91ab3cecefd87fc83b844aebc1" exitCode=0
Aug 04 18:46:17.038490 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.038373    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv" event={"ID":"78e7bc71-2810-49dd-83c8-2cc9997d28b1","Type":"ContainerDied","Data":"3899c86ec61131f8e184adcff0f458d557813a91ab3cecefd87fc83b844aebc1"}
Aug 04 18:46:17.081294 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.081270    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:46:17.248945 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.248911    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kube-rbac-proxy-sar-config\") pod \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") "
Aug 04 18:46:17.248945 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.248947    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78e7bc71-2810-49dd-83c8-2cc9997d28b1-cabundle-cert\") pod \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") "
Aug 04 18:46:17.249174 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.248965    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kserve-provision-location\") pod \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") "
Aug 04 18:46:17.249174 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.249010    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78e7bc71-2810-49dd-83c8-2cc9997d28b1-proxy-tls\") pod \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") "
Aug 04 18:46:17.249174 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.249059    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pb6f\" (UniqueName: \"kubernetes.io/projected/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kube-api-access-6pb6f\") pod \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\" (UID: \"78e7bc71-2810-49dd-83c8-2cc9997d28b1\") "
Aug 04 18:46:17.249392 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.249365    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "78e7bc71-2810-49dd-83c8-2cc9997d28b1" (UID: "78e7bc71-2810-49dd-83c8-2cc9997d28b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:46:17.249392 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.249379    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e7bc71-2810-49dd-83c8-2cc9997d28b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "78e7bc71-2810-49dd-83c8-2cc9997d28b1" (UID: "78e7bc71-2810-49dd-83c8-2cc9997d28b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:46:17.249533 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.249349    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "78e7bc71-2810-49dd-83c8-2cc9997d28b1" (UID: "78e7bc71-2810-49dd-83c8-2cc9997d28b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:46:17.251179 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.251149    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kube-api-access-6pb6f" (OuterVolumeSpecName: "kube-api-access-6pb6f") pod "78e7bc71-2810-49dd-83c8-2cc9997d28b1" (UID: "78e7bc71-2810-49dd-83c8-2cc9997d28b1"). InnerVolumeSpecName "kube-api-access-6pb6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:46:17.251295 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.251193    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e7bc71-2810-49dd-83c8-2cc9997d28b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "78e7bc71-2810-49dd-83c8-2cc9997d28b1" (UID: "78e7bc71-2810-49dd-83c8-2cc9997d28b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:46:17.349848 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.349815    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:46:17.349848 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.349843    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78e7bc71-2810-49dd-83c8-2cc9997d28b1-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:46:17.349848 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.349854    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:46:17.350090 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.349863    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78e7bc71-2810-49dd-83c8-2cc9997d28b1-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:46:17.350090 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:17.349872    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pb6f\" (UniqueName: \"kubernetes.io/projected/78e7bc71-2810-49dd-83c8-2cc9997d28b1-kube-api-access-6pb6f\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:46:18.045045 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:18.045006    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv" event={"ID":"78e7bc71-2810-49dd-83c8-2cc9997d28b1","Type":"ContainerDied","Data":"37fbc4b37d367e426411c0ed3eb03546061963cc42b8bb31a66db8105363d909"}
Aug 04 18:46:18.045495 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:18.045083    2566 scope.go:117] "RemoveContainer" containerID="f6a8df18a4123c604c20ea8e342cc872f275093346547526364b1469a08e3c15"
Aug 04 18:46:18.045495 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:18.045020    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"
Aug 04 18:46:18.054790 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:18.054768    2566 scope.go:117] "RemoveContainer" containerID="3899c86ec61131f8e184adcff0f458d557813a91ab3cecefd87fc83b844aebc1"
Aug 04 18:46:18.062256 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:18.062237    2566 scope.go:117] "RemoveContainer" containerID="14f7af37356bf19b6dc9683b9f154eadc0e12dcdd132a6981793dde745d06608"
Aug 04 18:46:18.075784 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:18.075760    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"]
Aug 04 18:46:18.080486 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:18.080462    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v5dfv"]
Aug 04 18:46:19.194314 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:19.194283    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" path="/var/lib/kubelet/pods/78e7bc71-2810-49dd-83c8-2cc9997d28b1/volumes"
Aug 04 18:46:37.975953 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:37.975921    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:46:44.345743 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:44.345704    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"]
Aug 04 18:46:44.346141 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:44.345946    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr" podUID="7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" containerName="kserve-container" containerID="cri-o://8a4443fd98ea2e2fa0a6176d7d43449afa285349a8b47bcacc686d2d4a2a2e71" gracePeriod=30
Aug 04 18:46:44.346141 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:44.345962    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr" podUID="7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" containerName="kube-rbac-proxy" containerID="cri-o://3c84e558c26ffcd02265a2e93671e46c5523fa13bf7faebf574240cfbe5166c8" gracePeriod=30
Aug 04 18:46:45.178087 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.178052    2566 generic.go:356] "Generic (PLEG): container finished" podID="7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" containerID="3c84e558c26ffcd02265a2e93671e46c5523fa13bf7faebf574240cfbe5166c8" exitCode=2
Aug 04 18:46:45.178266 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.178129    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr" event={"ID":"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165","Type":"ContainerDied","Data":"3c84e558c26ffcd02265a2e93671e46c5523fa13bf7faebf574240cfbe5166c8"}
Aug 04 18:46:45.609910 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.609888    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:46:45.712131 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.712040    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-cabundle-cert\") pod \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") "
Aug 04 18:46:45.712131 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.712102    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kube-rbac-proxy-sar-config\") pod \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") "
Aug 04 18:46:45.712131 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.712121    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kserve-provision-location\") pod \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") "
Aug 04 18:46:45.712474 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.712146    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-proxy-tls\") pod \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") "
Aug 04 18:46:45.712474 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.712175    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c72qb\" (UniqueName: \"kubernetes.io/projected/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kube-api-access-c72qb\") pod \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\" (UID: \"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165\") "
Aug 04 18:46:45.712591 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.712500    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" (UID: "7f36a27d-4808-4f14-b2ff-5c7f6c0b3165"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:46:45.712667 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.712638    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" (UID: "7f36a27d-4808-4f14-b2ff-5c7f6c0b3165"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:46:45.712714 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.712678    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" (UID: "7f36a27d-4808-4f14-b2ff-5c7f6c0b3165"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:46:45.714343 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.714323    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" (UID: "7f36a27d-4808-4f14-b2ff-5c7f6c0b3165"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:46:45.714467 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.714451    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kube-api-access-c72qb" (OuterVolumeSpecName: "kube-api-access-c72qb") pod "7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" (UID: "7f36a27d-4808-4f14-b2ff-5c7f6c0b3165"). InnerVolumeSpecName "kube-api-access-c72qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:46:45.813341 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.813302    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:46:45.813341 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.813336    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:46:45.813341 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.813347    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:46:45.813610 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.813377    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:46:45.813610 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:45.813385    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c72qb\" (UniqueName: \"kubernetes.io/projected/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165-kube-api-access-c72qb\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:46:46.185733 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:46.185698    2566 generic.go:356] "Generic (PLEG): container finished" podID="7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" containerID="8a4443fd98ea2e2fa0a6176d7d43449afa285349a8b47bcacc686d2d4a2a2e71" exitCode=0
Aug 04 18:46:46.185899 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:46.185764    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"
Aug 04 18:46:46.185899 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:46.185785    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr" event={"ID":"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165","Type":"ContainerDied","Data":"8a4443fd98ea2e2fa0a6176d7d43449afa285349a8b47bcacc686d2d4a2a2e71"}
Aug 04 18:46:46.185899 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:46.185851    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr" event={"ID":"7f36a27d-4808-4f14-b2ff-5c7f6c0b3165","Type":"ContainerDied","Data":"15f2f134c945df9c32b2f8f9069339e61fe855048c27c2316af0164ac1b8afaa"}
Aug 04 18:46:46.185899 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:46.185880    2566 scope.go:117] "RemoveContainer" containerID="3c84e558c26ffcd02265a2e93671e46c5523fa13bf7faebf574240cfbe5166c8"
Aug 04 18:46:46.194763 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:46.194746    2566 scope.go:117] "RemoveContainer" containerID="8a4443fd98ea2e2fa0a6176d7d43449afa285349a8b47bcacc686d2d4a2a2e71"
Aug 04 18:46:46.202306 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:46.202287    2566 scope.go:117] "RemoveContainer" containerID="a8d894a13c6997b49e2f39694caabbb6c8c60e6f0bb1eeede3b769bcbb96d47e"
Aug 04 18:46:46.208630 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:46.208607    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"]
Aug 04 18:46:46.210203 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:46.210188    2566 scope.go:117] "RemoveContainer" containerID="3c84e558c26ffcd02265a2e93671e46c5523fa13bf7faebf574240cfbe5166c8"
Aug 04 18:46:46.210504 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:46:46.210485    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c84e558c26ffcd02265a2e93671e46c5523fa13bf7faebf574240cfbe5166c8\": container with ID starting with 3c84e558c26ffcd02265a2e93671e46c5523fa13bf7faebf574240cfbe5166c8 not found: ID does not exist" containerID="3c84e558c26ffcd02265a2e93671e46c5523fa13bf7faebf574240cfbe5166c8"
Aug 04 18:46:46.210572 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:46.210514    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c84e558c26ffcd02265a2e93671e46c5523fa13bf7faebf574240cfbe5166c8"} err="failed to get container status \"3c84e558c26ffcd02265a2e93671e46c5523fa13bf7faebf574240cfbe5166c8\": rpc error: code = NotFound desc = could not find container \"3c84e558c26ffcd02265a2e93671e46c5523fa13bf7faebf574240cfbe5166c8\": container with ID starting with 3c84e558c26ffcd02265a2e93671e46c5523fa13bf7faebf574240cfbe5166c8 not found: ID does not exist"
Aug 04 18:46:46.210572 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:46.210532    2566 scope.go:117] "RemoveContainer" containerID="8a4443fd98ea2e2fa0a6176d7d43449afa285349a8b47bcacc686d2d4a2a2e71"
Aug 04 18:46:46.210772 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:46:46.210757    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4443fd98ea2e2fa0a6176d7d43449afa285349a8b47bcacc686d2d4a2a2e71\": container with ID starting with 8a4443fd98ea2e2fa0a6176d7d43449afa285349a8b47bcacc686d2d4a2a2e71 not found: ID does not exist" containerID="8a4443fd98ea2e2fa0a6176d7d43449afa285349a8b47bcacc686d2d4a2a2e71"
Aug 04 18:46:46.210827 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:46.210776    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4443fd98ea2e2fa0a6176d7d43449afa285349a8b47bcacc686d2d4a2a2e71"} err="failed to get container status \"8a4443fd98ea2e2fa0a6176d7d43449afa285349a8b47bcacc686d2d4a2a2e71\": rpc error: code = NotFound desc = could not find container \"8a4443fd98ea2e2fa0a6176d7d43449afa285349a8b47bcacc686d2d4a2a2e71\": container with ID starting with 8a4443fd98ea2e2fa0a6176d7d43449afa285349a8b47bcacc686d2d4a2a2e71 not found: ID does not exist"
Aug 04 18:46:46.210827 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:46.210796    2566 scope.go:117] "RemoveContainer" containerID="a8d894a13c6997b49e2f39694caabbb6c8c60e6f0bb1eeede3b769bcbb96d47e"
Aug 04 18:46:46.211281 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:46:46.211201    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d894a13c6997b49e2f39694caabbb6c8c60e6f0bb1eeede3b769bcbb96d47e\": container with ID starting with a8d894a13c6997b49e2f39694caabbb6c8c60e6f0bb1eeede3b769bcbb96d47e not found: ID does not exist" containerID="a8d894a13c6997b49e2f39694caabbb6c8c60e6f0bb1eeede3b769bcbb96d47e"
Aug 04 18:46:46.211281 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:46.211231    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d894a13c6997b49e2f39694caabbb6c8c60e6f0bb1eeede3b769bcbb96d47e"} err="failed to get container status \"a8d894a13c6997b49e2f39694caabbb6c8c60e6f0bb1eeede3b769bcbb96d47e\": rpc error: code = NotFound desc = could not find container \"a8d894a13c6997b49e2f39694caabbb6c8c60e6f0bb1eeede3b769bcbb96d47e\": container with ID starting with a8d894a13c6997b49e2f39694caabbb6c8c60e6f0bb1eeede3b769bcbb96d47e not found: ID does not exist"
Aug 04 18:46:46.213569 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:46.213549    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-lmbgr"]
Aug 04 18:46:47.194380 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:47.194322    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" path="/var/lib/kubelet/pods/7f36a27d-4808-4f14-b2ff-5c7f6c0b3165/volumes"
Aug 04 18:46:49.601457 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601420    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"]
Aug 04 18:46:49.601837 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601781    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerName="kserve-container"
Aug 04 18:46:49.601837 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601792    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerName="kserve-container"
Aug 04 18:46:49.601837 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601810    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" containerName="storage-initializer"
Aug 04 18:46:49.601837 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601815    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" containerName="storage-initializer"
Aug 04 18:46:49.601837 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601823    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerName="storage-initializer"
Aug 04 18:46:49.601837 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601829    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerName="storage-initializer"
Aug 04 18:46:49.601837 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601837    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" containerName="kube-rbac-proxy"
Aug 04 18:46:49.601837 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601843    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" containerName="kube-rbac-proxy"
Aug 04 18:46:49.602102 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601852    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" containerName="kserve-container"
Aug 04 18:46:49.602102 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601857    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" containerName="kserve-container"
Aug 04 18:46:49.602102 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601865    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerName="kube-rbac-proxy"
Aug 04 18:46:49.602102 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601870    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerName="kube-rbac-proxy"
Aug 04 18:46:49.602102 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601923    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerName="kserve-container"
Aug 04 18:46:49.602102 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601931    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" containerName="kube-rbac-proxy"
Aug 04 18:46:49.602102 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601937    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f36a27d-4808-4f14-b2ff-5c7f6c0b3165" containerName="kserve-container"
Aug 04 18:46:49.602102 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.601944    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e7bc71-2810-49dd-83c8-2cc9997d28b1" containerName="kube-rbac-proxy"
Aug 04 18:46:49.606875 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.606858    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:49.610957 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.610923    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:46:49.611110 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.611081    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:46:49.611601 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.611581    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:46:49.611601 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.611601    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Aug 04 18:46:49.612177 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.612162    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:46:49.612712 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.612687    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Aug 04 18:46:49.619366 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.619339    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:46:49.627598 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.627578    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"]
Aug 04 18:46:49.746964 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.746931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/085b365a-3293-4727-95bb-187095a8a2c0-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:49.746964 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.746967    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnqdw\" (UniqueName: \"kubernetes.io/projected/085b365a-3293-4727-95bb-187095a8a2c0-kube-api-access-xnqdw\") pod \"isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:49.747179 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.747046    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/085b365a-3293-4727-95bb-187095a8a2c0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:49.747179 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.747102    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/085b365a-3293-4727-95bb-187095a8a2c0-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:49.747179 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.747165    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/085b365a-3293-4727-95bb-187095a8a2c0-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:49.848320 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.848283    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/085b365a-3293-4727-95bb-187095a8a2c0-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:49.848320 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.848322    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnqdw\" (UniqueName: \"kubernetes.io/projected/085b365a-3293-4727-95bb-187095a8a2c0-kube-api-access-xnqdw\") pod \"isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:49.848569 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.848511    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/085b365a-3293-4727-95bb-187095a8a2c0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:49.848607 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.848570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/085b365a-3293-4727-95bb-187095a8a2c0-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:49.848644 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.848623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/085b365a-3293-4727-95bb-187095a8a2c0-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:49.849021 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.848999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/085b365a-3293-4727-95bb-187095a8a2c0-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:49.849167 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.849150    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/085b365a-3293-4727-95bb-187095a8a2c0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:49.849235 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.849210    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/085b365a-3293-4727-95bb-187095a8a2c0-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:49.850759 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.850738    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/085b365a-3293-4727-95bb-187095a8a2c0-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:49.867115 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.867036    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnqdw\" (UniqueName: \"kubernetes.io/projected/085b365a-3293-4727-95bb-187095a8a2c0-kube-api-access-xnqdw\") pod \"isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:49.917113 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:49.917076    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:50.062658 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:50.062625    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"]
Aug 04 18:46:50.065746 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:46:50.065718    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085b365a_3293_4727_95bb_187095a8a2c0.slice/crio-7ce2b09eb938c5194cf4298332fa47570277cf4635ffb914b3764d418fe6dc86 WatchSource:0}: Error finding container 7ce2b09eb938c5194cf4298332fa47570277cf4635ffb914b3764d418fe6dc86: Status 404 returned error can't find the container with id 7ce2b09eb938c5194cf4298332fa47570277cf4635ffb914b3764d418fe6dc86
Aug 04 18:46:50.208888 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:50.208855    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" event={"ID":"085b365a-3293-4727-95bb-187095a8a2c0","Type":"ContainerStarted","Data":"ef3d1c0c3a832fe565ee2dc4eb7a89722264805c40b2447b91ebd47545fb35f7"}
Aug 04 18:46:50.209060 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:50.208916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" event={"ID":"085b365a-3293-4727-95bb-187095a8a2c0","Type":"ContainerStarted","Data":"7ce2b09eb938c5194cf4298332fa47570277cf4635ffb914b3764d418fe6dc86"}
Aug 04 18:46:54.229066 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:54.229033    2566 generic.go:356] "Generic (PLEG): container finished" podID="085b365a-3293-4727-95bb-187095a8a2c0" containerID="ef3d1c0c3a832fe565ee2dc4eb7a89722264805c40b2447b91ebd47545fb35f7" exitCode=0
Aug 04 18:46:54.229492 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:54.229121    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" event={"ID":"085b365a-3293-4727-95bb-187095a8a2c0","Type":"ContainerDied","Data":"ef3d1c0c3a832fe565ee2dc4eb7a89722264805c40b2447b91ebd47545fb35f7"}
Aug 04 18:46:54.292012 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:54.291994    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:46:55.238807 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:55.238772    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" event={"ID":"085b365a-3293-4727-95bb-187095a8a2c0","Type":"ContainerStarted","Data":"6b78096a39e21fd1a70d101fd131425ca6b688fc6aea39485c0e87b0f6c7b829"}
Aug 04 18:46:57.253993 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:57.253962    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" event={"ID":"085b365a-3293-4727-95bb-187095a8a2c0","Type":"ContainerStarted","Data":"1b44f001dd9acffdb4d5b5348b878fe3b4dee44722575b45d1e6b659df273d27"}
Aug 04 18:46:58.263894 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:58.263852    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" event={"ID":"085b365a-3293-4727-95bb-187095a8a2c0","Type":"ContainerStarted","Data":"407df449bc82ff87f8f89862d04cfc10cdd566a3064f6b2fac95cdd6292dfbed"}
Aug 04 18:46:58.264401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:58.264069    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:58.264401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:58.264119    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:58.264401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:58.264142    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:46:58.303086 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:46:58.303032    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" podStartSLOduration=2.805966068 podStartE2EDuration="9.3030126s" totalImagesPullingTime="2.858332499s" totalInitContainerRuntime="3.638714033s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:46:58.301343578 +0000 UTC m=+2817.591957779" watchObservedRunningTime="2026-08-04 18:46:58.3030126 +0000 UTC m=+2817.593626771"
Aug 04 18:47:04.274415 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:47:04.274383    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:47:24.276079 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:47:24.276034    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.67:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.67:8080: connect: connection refused"
Aug 04 18:47:29.918068 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:47:29.918010    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.67:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.67:8080: connect: connection refused"
Aug 04 18:47:34.278770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:47:34.278739    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:48:04.279495 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:04.279464    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:48:09.617605 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:09.617570    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"]
Aug 04 18:48:09.618086 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:09.617849    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kserve-container" containerID="cri-o://6b78096a39e21fd1a70d101fd131425ca6b688fc6aea39485c0e87b0f6c7b829" gracePeriod=30
Aug 04 18:48:09.618086 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:09.617887    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kube-rbac-proxy" containerID="cri-o://407df449bc82ff87f8f89862d04cfc10cdd566a3064f6b2fac95cdd6292dfbed" gracePeriod=30
Aug 04 18:48:09.618086 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:09.617889    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kserve-agent" containerID="cri-o://1b44f001dd9acffdb4d5b5348b878fe3b4dee44722575b45d1e6b659df273d27" gracePeriod=30
Aug 04 18:48:10.586840 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:10.586805    2566 generic.go:356] "Generic (PLEG): container finished" podID="085b365a-3293-4727-95bb-187095a8a2c0" containerID="407df449bc82ff87f8f89862d04cfc10cdd566a3064f6b2fac95cdd6292dfbed" exitCode=2
Aug 04 18:48:10.587009 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:10.586875    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" event={"ID":"085b365a-3293-4727-95bb-187095a8a2c0","Type":"ContainerDied","Data":"407df449bc82ff87f8f89862d04cfc10cdd566a3064f6b2fac95cdd6292dfbed"}
Aug 04 18:48:12.600198 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:12.600155    2566 generic.go:356] "Generic (PLEG): container finished" podID="085b365a-3293-4727-95bb-187095a8a2c0" containerID="6b78096a39e21fd1a70d101fd131425ca6b688fc6aea39485c0e87b0f6c7b829" exitCode=0
Aug 04 18:48:12.600625 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:12.600210    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" event={"ID":"085b365a-3293-4727-95bb-187095a8a2c0","Type":"ContainerDied","Data":"6b78096a39e21fd1a70d101fd131425ca6b688fc6aea39485c0e87b0f6c7b829"}
Aug 04 18:48:14.269699 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:14.269645    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.67:8643/healthz\": dial tcp 10.134.0.67:8643: connect: connection refused"
Aug 04 18:48:14.275211 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:14.275166    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.67:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.67:8080: connect: connection refused"
Aug 04 18:48:19.270099 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:19.270050    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.67:8643/healthz\": dial tcp 10.134.0.67:8643: connect: connection refused"
Aug 04 18:48:24.269428 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:24.269385    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.67:8643/healthz\": dial tcp 10.134.0.67:8643: connect: connection refused"
Aug 04 18:48:24.269814 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:24.269503    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:48:24.275004 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:24.274977    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.67:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.67:8080: connect: connection refused"
Aug 04 18:48:29.269382 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:29.269311    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.67:8643/healthz\": dial tcp 10.134.0.67:8643: connect: connection refused"
Aug 04 18:48:34.269531 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:34.269487    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.67:8643/healthz\": dial tcp 10.134.0.67:8643: connect: connection refused"
Aug 04 18:48:34.274899 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:34.274864    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.67:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.67:8080: connect: connection refused"
Aug 04 18:48:34.275043 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:34.275026    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:48:39.270104 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:39.270056    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.67:8643/healthz\": dial tcp 10.134.0.67:8643: connect: connection refused"
Aug 04 18:48:39.720243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:39.720208    2566 generic.go:356] "Generic (PLEG): container finished" podID="085b365a-3293-4727-95bb-187095a8a2c0" containerID="1b44f001dd9acffdb4d5b5348b878fe3b4dee44722575b45d1e6b659df273d27" exitCode=137
Aug 04 18:48:39.720437 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:39.720266    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" event={"ID":"085b365a-3293-4727-95bb-187095a8a2c0","Type":"ContainerDied","Data":"1b44f001dd9acffdb4d5b5348b878fe3b4dee44722575b45d1e6b659df273d27"}
Aug 04 18:48:39.770566 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:39.770542    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:48:39.906507 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:39.906425    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/085b365a-3293-4727-95bb-187095a8a2c0-proxy-tls\") pod \"085b365a-3293-4727-95bb-187095a8a2c0\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") "
Aug 04 18:48:39.906507 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:39.906480    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/085b365a-3293-4727-95bb-187095a8a2c0-kserve-provision-location\") pod \"085b365a-3293-4727-95bb-187095a8a2c0\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") "
Aug 04 18:48:39.906729 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:39.906522    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnqdw\" (UniqueName: \"kubernetes.io/projected/085b365a-3293-4727-95bb-187095a8a2c0-kube-api-access-xnqdw\") pod \"085b365a-3293-4727-95bb-187095a8a2c0\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") "
Aug 04 18:48:39.906729 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:39.906545    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/085b365a-3293-4727-95bb-187095a8a2c0-kube-rbac-proxy-sar-config\") pod \"085b365a-3293-4727-95bb-187095a8a2c0\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") "
Aug 04 18:48:39.906729 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:39.906565    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/085b365a-3293-4727-95bb-187095a8a2c0-cabundle-cert\") pod \"085b365a-3293-4727-95bb-187095a8a2c0\" (UID: \"085b365a-3293-4727-95bb-187095a8a2c0\") "
Aug 04 18:48:39.906892 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:39.906868    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085b365a-3293-4727-95bb-187095a8a2c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "085b365a-3293-4727-95bb-187095a8a2c0" (UID: "085b365a-3293-4727-95bb-187095a8a2c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:48:39.906956 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:39.906912    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085b365a-3293-4727-95bb-187095a8a2c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "085b365a-3293-4727-95bb-187095a8a2c0" (UID: "085b365a-3293-4727-95bb-187095a8a2c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:48:39.906956 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:39.906937    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085b365a-3293-4727-95bb-187095a8a2c0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "085b365a-3293-4727-95bb-187095a8a2c0" (UID: "085b365a-3293-4727-95bb-187095a8a2c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:48:39.908722 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:39.908699    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/085b365a-3293-4727-95bb-187095a8a2c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "085b365a-3293-4727-95bb-187095a8a2c0" (UID: "085b365a-3293-4727-95bb-187095a8a2c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:48:39.908831 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:39.908758    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085b365a-3293-4727-95bb-187095a8a2c0-kube-api-access-xnqdw" (OuterVolumeSpecName: "kube-api-access-xnqdw") pod "085b365a-3293-4727-95bb-187095a8a2c0" (UID: "085b365a-3293-4727-95bb-187095a8a2c0"). InnerVolumeSpecName "kube-api-access-xnqdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:48:40.007553 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:40.007516    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnqdw\" (UniqueName: \"kubernetes.io/projected/085b365a-3293-4727-95bb-187095a8a2c0-kube-api-access-xnqdw\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:48:40.007553 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:40.007548    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/085b365a-3293-4727-95bb-187095a8a2c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:48:40.007553 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:40.007558    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/085b365a-3293-4727-95bb-187095a8a2c0-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:48:40.007804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:40.007569    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/085b365a-3293-4727-95bb-187095a8a2c0-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:48:40.007804 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:40.007578    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/085b365a-3293-4727-95bb-187095a8a2c0-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:48:40.727611 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:40.727574    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n" event={"ID":"085b365a-3293-4727-95bb-187095a8a2c0","Type":"ContainerDied","Data":"7ce2b09eb938c5194cf4298332fa47570277cf4635ffb914b3764d418fe6dc86"}
Aug 04 18:48:40.727611 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:40.727597    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"
Aug 04 18:48:40.728145 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:40.727648    2566 scope.go:117] "RemoveContainer" containerID="407df449bc82ff87f8f89862d04cfc10cdd566a3064f6b2fac95cdd6292dfbed"
Aug 04 18:48:40.737284 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:40.737266    2566 scope.go:117] "RemoveContainer" containerID="1b44f001dd9acffdb4d5b5348b878fe3b4dee44722575b45d1e6b659df273d27"
Aug 04 18:48:40.745188 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:40.745170    2566 scope.go:117] "RemoveContainer" containerID="6b78096a39e21fd1a70d101fd131425ca6b688fc6aea39485c0e87b0f6c7b829"
Aug 04 18:48:40.752160 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:40.752137    2566 scope.go:117] "RemoveContainer" containerID="ef3d1c0c3a832fe565ee2dc4eb7a89722264805c40b2447b91ebd47545fb35f7"
Aug 04 18:48:40.756667 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:40.756646    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"]
Aug 04 18:48:40.761239 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:40.761220    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67cc454cc7-xkn8n"]
Aug 04 18:48:41.194537 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:48:41.194506    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="085b365a-3293-4727-95bb-187095a8a2c0" path="/var/lib/kubelet/pods/085b365a-3293-4727-95bb-187095a8a2c0/volumes"
Aug 04 18:49:31.777965 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.777921    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"]
Aug 04 18:49:31.778508 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.778398    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="storage-initializer"
Aug 04 18:49:31.778508 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.778414    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="storage-initializer"
Aug 04 18:49:31.778508 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.778428    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kserve-agent"
Aug 04 18:49:31.778508 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.778434    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kserve-agent"
Aug 04 18:49:31.778508 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.778443    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kube-rbac-proxy"
Aug 04 18:49:31.778508 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.778449    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kube-rbac-proxy"
Aug 04 18:49:31.778508 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.778460    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kserve-container"
Aug 04 18:49:31.778508 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.778468    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kserve-container"
Aug 04 18:49:31.778826 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.778531    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kube-rbac-proxy"
Aug 04 18:49:31.778826 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.778539    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kserve-container"
Aug 04 18:49:31.778826 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.778548    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="085b365a-3293-4727-95bb-187095a8a2c0" containerName="kserve-agent"
Aug 04 18:49:31.782873 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.782853    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:31.793513 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.793490    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:49:31.794403 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.794382    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Aug 04 18:49:31.794482 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.794460    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:49:31.794848 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.794828    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Aug 04 18:49:31.795969 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.795951    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:49:31.796068 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.796049    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:49:31.796395 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.796347    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:49:31.803157 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.803104    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"]
Aug 04 18:49:31.855774 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.855744    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp8xb\" (UniqueName: \"kubernetes.io/projected/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kube-api-access-gp8xb\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b779x\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:31.855965 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.855804    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b779x\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:31.855965 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.855829    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b779x\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:31.855965 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.855908    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b779x\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:31.856114 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.855975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b779x\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:31.957308 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.957278    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp8xb\" (UniqueName: \"kubernetes.io/projected/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kube-api-access-gp8xb\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b779x\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:31.957514 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.957342    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b779x\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:31.957514 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.957397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b779x\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:31.957514 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.957433    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b779x\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:31.957514 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.957484    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b779x\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:31.957793 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.957770    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b779x\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:31.958037 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.958012    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b779x\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:31.958143 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.958051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b779x\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:31.959901 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.959873    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b779x\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:31.967692 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:31.967670    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp8xb\" (UniqueName: \"kubernetes.io/projected/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kube-api-access-gp8xb\") pod \"isvc-paddle-runtime-predictor-58b99fff76-b779x\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:32.092836 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:32.092800    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:32.239170 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:32.239146    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"]
Aug 04 18:49:32.241316 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:49:32.241285    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca2e1dd_aad8_4583_a5b2_190b9e9eee64.slice/crio-84ee046bd74e325c41ab5502429ce5ce8d23749f0723c1c5b3298e6b9a7dbac7 WatchSource:0}: Error finding container 84ee046bd74e325c41ab5502429ce5ce8d23749f0723c1c5b3298e6b9a7dbac7: Status 404 returned error can't find the container with id 84ee046bd74e325c41ab5502429ce5ce8d23749f0723c1c5b3298e6b9a7dbac7
Aug 04 18:49:32.961841 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:32.961806    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" event={"ID":"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64","Type":"ContainerStarted","Data":"88e381a83e18e3822f43c97d26d1990b4897ce3c1a04e2ab45adabdfe3ca4202"}
Aug 04 18:49:32.962215 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:32.961864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" event={"ID":"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64","Type":"ContainerStarted","Data":"84ee046bd74e325c41ab5502429ce5ce8d23749f0723c1c5b3298e6b9a7dbac7"}
Aug 04 18:49:37.985166 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:37.985132    2566 generic.go:356] "Generic (PLEG): container finished" podID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerID="88e381a83e18e3822f43c97d26d1990b4897ce3c1a04e2ab45adabdfe3ca4202" exitCode=0
Aug 04 18:49:37.985726 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:37.985211    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" event={"ID":"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64","Type":"ContainerDied","Data":"88e381a83e18e3822f43c97d26d1990b4897ce3c1a04e2ab45adabdfe3ca4202"}
Aug 04 18:49:50.052343 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:50.052305    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" event={"ID":"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64","Type":"ContainerStarted","Data":"4f8adc1b9b04d48059ac0adbd23e2b87de213921a32d7de90dc63e2db252981d"}
Aug 04 18:49:50.052912 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:50.052410    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" event={"ID":"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64","Type":"ContainerStarted","Data":"2644266efa552790a5848c1e3677cb3a158906301bfecf262a060a95a7486c92"}
Aug 04 18:49:50.052912 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:50.052705    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:50.052912 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:50.052759    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:50.053992 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:50.053967    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Aug 04 18:49:50.075724 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:50.075675    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" podStartSLOduration=2.984255441 podStartE2EDuration="19.07566206s" totalImagesPullingTime="11.324330516s" totalInitContainerRuntime="4.767076103s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:49:50.074836192 +0000 UTC m=+2989.365450390" watchObservedRunningTime="2026-08-04 18:49:50.07566206 +0000 UTC m=+2989.366276230"
Aug 04 18:49:51.060348 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:51.060304    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Aug 04 18:49:56.065068 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:56.065038    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:49:56.065568 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:49:56.065541    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Aug 04 18:50:06.065547 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:06.065504    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Aug 04 18:50:16.066067 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:16.066025    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Aug 04 18:50:26.066197 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:26.066155    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Aug 04 18:50:36.066527 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:36.066495    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:50:43.327236 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:43.327202    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"]
Aug 04 18:50:43.327635 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:43.327513    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="kserve-container" containerID="cri-o://2644266efa552790a5848c1e3677cb3a158906301bfecf262a060a95a7486c92" gracePeriod=30
Aug 04 18:50:43.327635 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:43.327530    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="kube-rbac-proxy" containerID="cri-o://4f8adc1b9b04d48059ac0adbd23e2b87de213921a32d7de90dc63e2db252981d" gracePeriod=30
Aug 04 18:50:44.297491 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:44.297458    2566 generic.go:356] "Generic (PLEG): container finished" podID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerID="4f8adc1b9b04d48059ac0adbd23e2b87de213921a32d7de90dc63e2db252981d" exitCode=2
Aug 04 18:50:44.297663 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:44.297490    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" event={"ID":"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64","Type":"ContainerDied","Data":"4f8adc1b9b04d48059ac0adbd23e2b87de213921a32d7de90dc63e2db252981d"}
Aug 04 18:50:46.061086 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.061039    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.68:8643/healthz\": dial tcp 10.134.0.68:8643: connect: connection refused"
Aug 04 18:50:46.066532 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.066499    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Aug 04 18:50:46.175237 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.175215    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:50:46.221059 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.220981    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kube-rbac-proxy-sar-config\") pod \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") "
Aug 04 18:50:46.221059 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.221045    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kserve-provision-location\") pod \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") "
Aug 04 18:50:46.221259 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.221084    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-proxy-tls\") pod \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") "
Aug 04 18:50:46.221259 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.221133    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp8xb\" (UniqueName: \"kubernetes.io/projected/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kube-api-access-gp8xb\") pod \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") "
Aug 04 18:50:46.221259 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.221188    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-cabundle-cert\") pod \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\" (UID: \"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64\") "
Aug 04 18:50:46.221463 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.221347    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" (UID: "1ca2e1dd-aad8-4583-a5b2-190b9e9eee64"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:50:46.221525 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.221492    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:50:46.221608 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.221581    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" (UID: "1ca2e1dd-aad8-4583-a5b2-190b9e9eee64"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:50:46.223280 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.223260    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" (UID: "1ca2e1dd-aad8-4583-a5b2-190b9e9eee64"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:50:46.223377 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.223344    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kube-api-access-gp8xb" (OuterVolumeSpecName: "kube-api-access-gp8xb") pod "1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" (UID: "1ca2e1dd-aad8-4583-a5b2-190b9e9eee64"). InnerVolumeSpecName "kube-api-access-gp8xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:50:46.231273 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.231249    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" (UID: "1ca2e1dd-aad8-4583-a5b2-190b9e9eee64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:50:46.308652 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.308609    2566 generic.go:356] "Generic (PLEG): container finished" podID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerID="2644266efa552790a5848c1e3677cb3a158906301bfecf262a060a95a7486c92" exitCode=0
Aug 04 18:50:46.308819 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.308686    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"
Aug 04 18:50:46.308819 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.308686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" event={"ID":"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64","Type":"ContainerDied","Data":"2644266efa552790a5848c1e3677cb3a158906301bfecf262a060a95a7486c92"}
Aug 04 18:50:46.308819 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.308816    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x" event={"ID":"1ca2e1dd-aad8-4583-a5b2-190b9e9eee64","Type":"ContainerDied","Data":"84ee046bd74e325c41ab5502429ce5ce8d23749f0723c1c5b3298e6b9a7dbac7"}
Aug 04 18:50:46.308955 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.308845    2566 scope.go:117] "RemoveContainer" containerID="4f8adc1b9b04d48059ac0adbd23e2b87de213921a32d7de90dc63e2db252981d"
Aug 04 18:50:46.317428 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.317409    2566 scope.go:117] "RemoveContainer" containerID="2644266efa552790a5848c1e3677cb3a158906301bfecf262a060a95a7486c92"
Aug 04 18:50:46.322644 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.322623    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:50:46.322644 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.322645    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:50:46.322765 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.322655    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:50:46.322765 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.322663    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp8xb\" (UniqueName: \"kubernetes.io/projected/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64-kube-api-access-gp8xb\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:50:46.325033 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.325015    2566 scope.go:117] "RemoveContainer" containerID="88e381a83e18e3822f43c97d26d1990b4897ce3c1a04e2ab45adabdfe3ca4202"
Aug 04 18:50:46.332623 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.332603    2566 scope.go:117] "RemoveContainer" containerID="4f8adc1b9b04d48059ac0adbd23e2b87de213921a32d7de90dc63e2db252981d"
Aug 04 18:50:46.332886 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:50:46.332863    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8adc1b9b04d48059ac0adbd23e2b87de213921a32d7de90dc63e2db252981d\": container with ID starting with 4f8adc1b9b04d48059ac0adbd23e2b87de213921a32d7de90dc63e2db252981d not found: ID does not exist" containerID="4f8adc1b9b04d48059ac0adbd23e2b87de213921a32d7de90dc63e2db252981d"
Aug 04 18:50:46.332944 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.332900    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8adc1b9b04d48059ac0adbd23e2b87de213921a32d7de90dc63e2db252981d"} err="failed to get container status \"4f8adc1b9b04d48059ac0adbd23e2b87de213921a32d7de90dc63e2db252981d\": rpc error: code = NotFound desc = could not find container \"4f8adc1b9b04d48059ac0adbd23e2b87de213921a32d7de90dc63e2db252981d\": container with ID starting with 4f8adc1b9b04d48059ac0adbd23e2b87de213921a32d7de90dc63e2db252981d not found: ID does not exist"
Aug 04 18:50:46.332944 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.332925    2566 scope.go:117] "RemoveContainer" containerID="2644266efa552790a5848c1e3677cb3a158906301bfecf262a060a95a7486c92"
Aug 04 18:50:46.333188 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:50:46.333170    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2644266efa552790a5848c1e3677cb3a158906301bfecf262a060a95a7486c92\": container with ID starting with 2644266efa552790a5848c1e3677cb3a158906301bfecf262a060a95a7486c92 not found: ID does not exist" containerID="2644266efa552790a5848c1e3677cb3a158906301bfecf262a060a95a7486c92"
Aug 04 18:50:46.333229 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.333195    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2644266efa552790a5848c1e3677cb3a158906301bfecf262a060a95a7486c92"} err="failed to get container status \"2644266efa552790a5848c1e3677cb3a158906301bfecf262a060a95a7486c92\": rpc error: code = NotFound desc = could not find container \"2644266efa552790a5848c1e3677cb3a158906301bfecf262a060a95a7486c92\": container with ID starting with 2644266efa552790a5848c1e3677cb3a158906301bfecf262a060a95a7486c92 not found: ID does not exist"
Aug 04 18:50:46.333229 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.333211    2566 scope.go:117] "RemoveContainer" containerID="88e381a83e18e3822f43c97d26d1990b4897ce3c1a04e2ab45adabdfe3ca4202"
Aug 04 18:50:46.333591 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:50:46.333574    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e381a83e18e3822f43c97d26d1990b4897ce3c1a04e2ab45adabdfe3ca4202\": container with ID starting with 88e381a83e18e3822f43c97d26d1990b4897ce3c1a04e2ab45adabdfe3ca4202 not found: ID does not exist" containerID="88e381a83e18e3822f43c97d26d1990b4897ce3c1a04e2ab45adabdfe3ca4202"
Aug 04 18:50:46.333652 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.333595    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e381a83e18e3822f43c97d26d1990b4897ce3c1a04e2ab45adabdfe3ca4202"} err="failed to get container status \"88e381a83e18e3822f43c97d26d1990b4897ce3c1a04e2ab45adabdfe3ca4202\": rpc error: code = NotFound desc = could not find container \"88e381a83e18e3822f43c97d26d1990b4897ce3c1a04e2ab45adabdfe3ca4202\": container with ID starting with 88e381a83e18e3822f43c97d26d1990b4897ce3c1a04e2ab45adabdfe3ca4202 not found: ID does not exist"
Aug 04 18:50:46.333819 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.333803    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"]
Aug 04 18:50:46.338740 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:46.338718    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-b779x"]
Aug 04 18:50:47.193687 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:47.193655    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" path="/var/lib/kubelet/pods/1ca2e1dd-aad8-4583-a5b2-190b9e9eee64/volumes"
Aug 04 18:50:48.498012 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.497976    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"]
Aug 04 18:50:48.498478 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.498459    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="storage-initializer"
Aug 04 18:50:48.498531 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.498484    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="storage-initializer"
Aug 04 18:50:48.498531 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.498508    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="kserve-container"
Aug 04 18:50:48.498531 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.498517    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="kserve-container"
Aug 04 18:50:48.498626 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.498536    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="kube-rbac-proxy"
Aug 04 18:50:48.498626 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.498545    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="kube-rbac-proxy"
Aug 04 18:50:48.498626 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.498607    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="kube-rbac-proxy"
Aug 04 18:50:48.498626 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.498620    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ca2e1dd-aad8-4583-a5b2-190b9e9eee64" containerName="kserve-container"
Aug 04 18:50:48.501785 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.501769    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:48.504309 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.504286    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Aug 04 18:50:48.504309 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.504299    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:50:48.504601 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.504587    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Aug 04 18:50:48.504662 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.504651    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:50:48.504725 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.504677    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:50:48.505278 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.505256    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:50:48.505378 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.505345    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:50:48.514568 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.514543    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"]
Aug 04 18:50:48.545218 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.539594    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/405f8257-98e3-4618-bf0e-cbd4e936b7e3-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:48.545218 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.539710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg5qc\" (UniqueName: \"kubernetes.io/projected/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kube-api-access-vg5qc\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:48.545218 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.539856    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:48.545218 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.539902    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/405f8257-98e3-4618-bf0e-cbd4e936b7e3-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:48.545218 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.539968    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:48.640963 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.640928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg5qc\" (UniqueName: \"kubernetes.io/projected/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kube-api-access-vg5qc\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:48.641169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.641011    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:48.641169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.641031    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/405f8257-98e3-4618-bf0e-cbd4e936b7e3-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:48.641169 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.641064    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:48.641374 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:50:48.641192    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Aug 04 18:50:48.641374 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.641197    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/405f8257-98e3-4618-bf0e-cbd4e936b7e3-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:48.641374 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:50:48.641262    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/405f8257-98e3-4618-bf0e-cbd4e936b7e3-proxy-tls podName:405f8257-98e3-4618-bf0e-cbd4e936b7e3 nodeName:}" failed. No retries permitted until 2026-08-04 18:50:49.141239956 +0000 UTC m=+3048.431854111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/405f8257-98e3-4618-bf0e-cbd4e936b7e3-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" (UID: "405f8257-98e3-4618-bf0e-cbd4e936b7e3") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Aug 04 18:50:48.641520 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.641505    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:48.641784 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.641765    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/405f8257-98e3-4618-bf0e-cbd4e936b7e3-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:48.641853 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.641836    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:48.655293 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:48.655258    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg5qc\" (UniqueName: \"kubernetes.io/projected/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kube-api-access-vg5qc\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:49.145059 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:49.145018    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/405f8257-98e3-4618-bf0e-cbd4e936b7e3-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:49.147434 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:49.147413    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/405f8257-98e3-4618-bf0e-cbd4e936b7e3-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:49.412892 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:49.412806    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:49.548099 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:49.548077    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"]
Aug 04 18:50:49.550109 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:50:49.550082    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405f8257_98e3_4618_bf0e_cbd4e936b7e3.slice/crio-9fa230e6eef2e5aa2ef8715b0ded9668f1718fe984d41052c5bfa1f0600b3865 WatchSource:0}: Error finding container 9fa230e6eef2e5aa2ef8715b0ded9668f1718fe984d41052c5bfa1f0600b3865: Status 404 returned error can't find the container with id 9fa230e6eef2e5aa2ef8715b0ded9668f1718fe984d41052c5bfa1f0600b3865
Aug 04 18:50:50.328585 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:50.328542    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" event={"ID":"405f8257-98e3-4618-bf0e-cbd4e936b7e3","Type":"ContainerStarted","Data":"aaa0da86b41c2780542ac1ebe73a324ce9820b7533f3fd7e737ec44cdc960153"}
Aug 04 18:50:50.328750 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:50.328616    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" event={"ID":"405f8257-98e3-4618-bf0e-cbd4e936b7e3","Type":"ContainerStarted","Data":"9fa230e6eef2e5aa2ef8715b0ded9668f1718fe984d41052c5bfa1f0600b3865"}
Aug 04 18:50:54.350072 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:54.350041    2566 generic.go:356] "Generic (PLEG): container finished" podID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerID="aaa0da86b41c2780542ac1ebe73a324ce9820b7533f3fd7e737ec44cdc960153" exitCode=0
Aug 04 18:50:54.350072 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:54.350076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" event={"ID":"405f8257-98e3-4618-bf0e-cbd4e936b7e3","Type":"ContainerDied","Data":"aaa0da86b41c2780542ac1ebe73a324ce9820b7533f3fd7e737ec44cdc960153"}
Aug 04 18:50:55.357731 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:55.357692    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" event={"ID":"405f8257-98e3-4618-bf0e-cbd4e936b7e3","Type":"ContainerStarted","Data":"e8a67bdcd5e9740bf7895e655ee4870e029dd23c5b06e775308ad5f7bfe7e551"}
Aug 04 18:50:55.358223 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:55.357766    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" event={"ID":"405f8257-98e3-4618-bf0e-cbd4e936b7e3","Type":"ContainerStarted","Data":"68fe0036b4352bf5540e41cc0b2ff3fad6293a350841efd023fe836c0fef7767"}
Aug 04 18:50:55.358223 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:55.357994    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:55.388461 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:55.388381    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" podStartSLOduration=2.672441419 podStartE2EDuration="7.388345732s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.715904313s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:50:55.386833656 +0000 UTC m=+3054.677447839" watchObservedRunningTime="2026-08-04 18:50:55.388345732 +0000 UTC m=+3054.678959969"
Aug 04 18:50:56.363592 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:56.363559    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:50:56.364851 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:56.364827    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Aug 04 18:50:57.368482 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:50:57.368444    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Aug 04 18:51:02.372937 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:02.372908    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:51:02.373530 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:02.373502    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Aug 04 18:51:12.374187 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:12.374147    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Aug 04 18:51:22.374318 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:22.374274    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Aug 04 18:51:32.373816 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:32.373779    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Aug 04 18:51:42.374445 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:42.374416    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:51:50.414031 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:50.413998    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"]
Aug 04 18:51:50.414522 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:50.414302    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="kserve-container" containerID="cri-o://68fe0036b4352bf5540e41cc0b2ff3fad6293a350841efd023fe836c0fef7767" gracePeriod=30
Aug 04 18:51:50.414522 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:50.414374    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="kube-rbac-proxy" containerID="cri-o://e8a67bdcd5e9740bf7895e655ee4870e029dd23c5b06e775308ad5f7bfe7e551" gracePeriod=30
Aug 04 18:51:50.619294 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:50.619255    2566 generic.go:356] "Generic (PLEG): container finished" podID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerID="e8a67bdcd5e9740bf7895e655ee4870e029dd23c5b06e775308ad5f7bfe7e551" exitCode=2
Aug 04 18:51:50.619294 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:50.619295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" event={"ID":"405f8257-98e3-4618-bf0e-cbd4e936b7e3","Type":"ContainerDied","Data":"e8a67bdcd5e9740bf7895e655ee4870e029dd23c5b06e775308ad5f7bfe7e551"}
Aug 04 18:51:52.369657 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:52.369611    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.69:8643/healthz\": dial tcp 10.134.0.69:8643: connect: connection refused"
Aug 04 18:51:52.374255 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:52.374224    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Aug 04 18:51:53.260262 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.260239    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:51:53.332034 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.331934    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/405f8257-98e3-4618-bf0e-cbd4e936b7e3-proxy-tls\") pod \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") "
Aug 04 18:51:53.332034 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.332001    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg5qc\" (UniqueName: \"kubernetes.io/projected/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kube-api-access-vg5qc\") pod \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") "
Aug 04 18:51:53.332034 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.332018    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kube-rbac-proxy-sar-config\") pod \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") "
Aug 04 18:51:53.332319 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.332084    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kserve-provision-location\") pod \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") "
Aug 04 18:51:53.332319 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.332107    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/405f8257-98e3-4618-bf0e-cbd4e936b7e3-cabundle-cert\") pod \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\" (UID: \"405f8257-98e3-4618-bf0e-cbd4e936b7e3\") "
Aug 04 18:51:53.332512 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.332488    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "405f8257-98e3-4618-bf0e-cbd4e936b7e3" (UID: "405f8257-98e3-4618-bf0e-cbd4e936b7e3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:51:53.332554 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.332494    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405f8257-98e3-4618-bf0e-cbd4e936b7e3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "405f8257-98e3-4618-bf0e-cbd4e936b7e3" (UID: "405f8257-98e3-4618-bf0e-cbd4e936b7e3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:51:53.334222 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.334199    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kube-api-access-vg5qc" (OuterVolumeSpecName: "kube-api-access-vg5qc") pod "405f8257-98e3-4618-bf0e-cbd4e936b7e3" (UID: "405f8257-98e3-4618-bf0e-cbd4e936b7e3"). InnerVolumeSpecName "kube-api-access-vg5qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:51:53.334530 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.334505    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405f8257-98e3-4618-bf0e-cbd4e936b7e3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "405f8257-98e3-4618-bf0e-cbd4e936b7e3" (UID: "405f8257-98e3-4618-bf0e-cbd4e936b7e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:51:53.342805 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.342780    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "405f8257-98e3-4618-bf0e-cbd4e936b7e3" (UID: "405f8257-98e3-4618-bf0e-cbd4e936b7e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:51:53.433251 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.433220    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:51:53.433251 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.433246    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/405f8257-98e3-4618-bf0e-cbd4e936b7e3-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:51:53.433251 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.433257    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/405f8257-98e3-4618-bf0e-cbd4e936b7e3-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:51:53.433251 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.433264    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg5qc\" (UniqueName: \"kubernetes.io/projected/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kube-api-access-vg5qc\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:51:53.433740 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.433274    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/405f8257-98e3-4618-bf0e-cbd4e936b7e3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:51:53.633991 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.633911    2566 generic.go:356] "Generic (PLEG): container finished" podID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerID="68fe0036b4352bf5540e41cc0b2ff3fad6293a350841efd023fe836c0fef7767" exitCode=0
Aug 04 18:51:53.633991 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.633958    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" event={"ID":"405f8257-98e3-4618-bf0e-cbd4e936b7e3","Type":"ContainerDied","Data":"68fe0036b4352bf5540e41cc0b2ff3fad6293a350841efd023fe836c0fef7767"}
Aug 04 18:51:53.633991 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.633966    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"
Aug 04 18:51:53.634219 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.634006    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2" event={"ID":"405f8257-98e3-4618-bf0e-cbd4e936b7e3","Type":"ContainerDied","Data":"9fa230e6eef2e5aa2ef8715b0ded9668f1718fe984d41052c5bfa1f0600b3865"}
Aug 04 18:51:53.634219 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.634034    2566 scope.go:117] "RemoveContainer" containerID="e8a67bdcd5e9740bf7895e655ee4870e029dd23c5b06e775308ad5f7bfe7e551"
Aug 04 18:51:53.646438 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.646419    2566 scope.go:117] "RemoveContainer" containerID="68fe0036b4352bf5540e41cc0b2ff3fad6293a350841efd023fe836c0fef7767"
Aug 04 18:51:53.653857 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.653840    2566 scope.go:117] "RemoveContainer" containerID="aaa0da86b41c2780542ac1ebe73a324ce9820b7533f3fd7e737ec44cdc960153"
Aug 04 18:51:53.661307 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.661289    2566 scope.go:117] "RemoveContainer" containerID="e8a67bdcd5e9740bf7895e655ee4870e029dd23c5b06e775308ad5f7bfe7e551"
Aug 04 18:51:53.661570 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:51:53.661553    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a67bdcd5e9740bf7895e655ee4870e029dd23c5b06e775308ad5f7bfe7e551\": container with ID starting with e8a67bdcd5e9740bf7895e655ee4870e029dd23c5b06e775308ad5f7bfe7e551 not found: ID does not exist" containerID="e8a67bdcd5e9740bf7895e655ee4870e029dd23c5b06e775308ad5f7bfe7e551"
Aug 04 18:51:53.661634 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.661578    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a67bdcd5e9740bf7895e655ee4870e029dd23c5b06e775308ad5f7bfe7e551"} err="failed to get container status \"e8a67bdcd5e9740bf7895e655ee4870e029dd23c5b06e775308ad5f7bfe7e551\": rpc error: code = NotFound desc = could not find container \"e8a67bdcd5e9740bf7895e655ee4870e029dd23c5b06e775308ad5f7bfe7e551\": container with ID starting with e8a67bdcd5e9740bf7895e655ee4870e029dd23c5b06e775308ad5f7bfe7e551 not found: ID does not exist"
Aug 04 18:51:53.661634 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.661596    2566 scope.go:117] "RemoveContainer" containerID="68fe0036b4352bf5540e41cc0b2ff3fad6293a350841efd023fe836c0fef7767"
Aug 04 18:51:53.661827 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:51:53.661813    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68fe0036b4352bf5540e41cc0b2ff3fad6293a350841efd023fe836c0fef7767\": container with ID starting with 68fe0036b4352bf5540e41cc0b2ff3fad6293a350841efd023fe836c0fef7767 not found: ID does not exist" containerID="68fe0036b4352bf5540e41cc0b2ff3fad6293a350841efd023fe836c0fef7767"
Aug 04 18:51:53.661864 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.661833    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68fe0036b4352bf5540e41cc0b2ff3fad6293a350841efd023fe836c0fef7767"} err="failed to get container status \"68fe0036b4352bf5540e41cc0b2ff3fad6293a350841efd023fe836c0fef7767\": rpc error: code = NotFound desc = could not find container \"68fe0036b4352bf5540e41cc0b2ff3fad6293a350841efd023fe836c0fef7767\": container with ID starting with 68fe0036b4352bf5540e41cc0b2ff3fad6293a350841efd023fe836c0fef7767 not found: ID does not exist"
Aug 04 18:51:53.661864 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.661853    2566 scope.go:117] "RemoveContainer" containerID="aaa0da86b41c2780542ac1ebe73a324ce9820b7533f3fd7e737ec44cdc960153"
Aug 04 18:51:53.662093 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:51:53.662075    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa0da86b41c2780542ac1ebe73a324ce9820b7533f3fd7e737ec44cdc960153\": container with ID starting with aaa0da86b41c2780542ac1ebe73a324ce9820b7533f3fd7e737ec44cdc960153 not found: ID does not exist" containerID="aaa0da86b41c2780542ac1ebe73a324ce9820b7533f3fd7e737ec44cdc960153"
Aug 04 18:51:53.662141 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.662100    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa0da86b41c2780542ac1ebe73a324ce9820b7533f3fd7e737ec44cdc960153"} err="failed to get container status \"aaa0da86b41c2780542ac1ebe73a324ce9820b7533f3fd7e737ec44cdc960153\": rpc error: code = NotFound desc = could not find container \"aaa0da86b41c2780542ac1ebe73a324ce9820b7533f3fd7e737ec44cdc960153\": container with ID starting with aaa0da86b41c2780542ac1ebe73a324ce9820b7533f3fd7e737ec44cdc960153 not found: ID does not exist"
Aug 04 18:51:53.677783 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.677753    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"]
Aug 04 18:51:53.684626 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:53.684593    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-btdj2"]
Aug 04 18:51:55.194458 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:51:55.194429    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" path="/var/lib/kubelet/pods/405f8257-98e3-4618-bf0e-cbd4e936b7e3/volumes"
Aug 04 18:53:42.174171 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.174131    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"]
Aug 04 18:53:42.174661 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.174522    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="kube-rbac-proxy"
Aug 04 18:53:42.174661 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.174535    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="kube-rbac-proxy"
Aug 04 18:53:42.174661 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.174543    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="kserve-container"
Aug 04 18:53:42.174661 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.174549    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="kserve-container"
Aug 04 18:53:42.174661 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.174570    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="storage-initializer"
Aug 04 18:53:42.174661 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.174580    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="storage-initializer"
Aug 04 18:53:42.174661 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.174639    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="kserve-container"
Aug 04 18:53:42.174661 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.174650    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="405f8257-98e3-4618-bf0e-cbd4e936b7e3" containerName="kube-rbac-proxy"
Aug 04 18:53:42.177072 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.177052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.180083 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.180063    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Aug 04 18:53:42.180478 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.180454    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:53:42.180600 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.180557    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:53:42.181174 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.181146    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wc562\""
Aug 04 18:53:42.181282 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.181173    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:53:42.181282 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.181191    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Aug 04 18:53:42.181282 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.181260    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:53:42.195892 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.195870    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"]
Aug 04 18:53:42.289752 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.289717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwb8h\" (UniqueName: \"kubernetes.io/projected/77e4ca22-8569-4063-aa66-e6dd914164fe-kube-api-access-kwb8h\") pod \"isvc-pmml-runtime-predictor-885fc8777-9krkw\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.289752 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.289752    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77e4ca22-8569-4063-aa66-e6dd914164fe-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-9krkw\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.289960 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.289850    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77e4ca22-8569-4063-aa66-e6dd914164fe-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-9krkw\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.289960 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.289875    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77e4ca22-8569-4063-aa66-e6dd914164fe-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-9krkw\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.289960 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.289897    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77e4ca22-8569-4063-aa66-e6dd914164fe-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-9krkw\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.390968 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.390930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwb8h\" (UniqueName: \"kubernetes.io/projected/77e4ca22-8569-4063-aa66-e6dd914164fe-kube-api-access-kwb8h\") pod \"isvc-pmml-runtime-predictor-885fc8777-9krkw\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.390968 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.390968    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77e4ca22-8569-4063-aa66-e6dd914164fe-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-9krkw\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.391178 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.391023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77e4ca22-8569-4063-aa66-e6dd914164fe-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-9krkw\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.391178 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.391049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77e4ca22-8569-4063-aa66-e6dd914164fe-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-9krkw\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.391178 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.391070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77e4ca22-8569-4063-aa66-e6dd914164fe-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-9krkw\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.391590 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.391552    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77e4ca22-8569-4063-aa66-e6dd914164fe-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-9krkw\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.391874 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.391853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77e4ca22-8569-4063-aa66-e6dd914164fe-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-9krkw\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.391874 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.391866    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77e4ca22-8569-4063-aa66-e6dd914164fe-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-9krkw\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.393547 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.393528    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77e4ca22-8569-4063-aa66-e6dd914164fe-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-9krkw\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.404064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.404039    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwb8h\" (UniqueName: \"kubernetes.io/projected/77e4ca22-8569-4063-aa66-e6dd914164fe-kube-api-access-kwb8h\") pod \"isvc-pmml-runtime-predictor-885fc8777-9krkw\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.487709 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.487618    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:42.625624 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:42.625537    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"]
Aug 04 18:53:42.627899 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:53:42.627867    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e4ca22_8569_4063_aa66_e6dd914164fe.slice/crio-fe61695b530bbc9bc750e9efcb9be28cf9a6de3c9f9cb8f42506e4699d5c6515 WatchSource:0}: Error finding container fe61695b530bbc9bc750e9efcb9be28cf9a6de3c9f9cb8f42506e4699d5c6515: Status 404 returned error can't find the container with id fe61695b530bbc9bc750e9efcb9be28cf9a6de3c9f9cb8f42506e4699d5c6515
Aug 04 18:53:43.111934 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:43.111898    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" event={"ID":"77e4ca22-8569-4063-aa66-e6dd914164fe","Type":"ContainerStarted","Data":"743e66d50c1c2700b3c3310aabe96bb5338f7f01ca064c17d7f5c77736622dcc"}
Aug 04 18:53:43.112114 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:43.111967    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" event={"ID":"77e4ca22-8569-4063-aa66-e6dd914164fe","Type":"ContainerStarted","Data":"fe61695b530bbc9bc750e9efcb9be28cf9a6de3c9f9cb8f42506e4699d5c6515"}
Aug 04 18:53:47.131128 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:47.131096    2566 generic.go:356] "Generic (PLEG): container finished" podID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerID="743e66d50c1c2700b3c3310aabe96bb5338f7f01ca064c17d7f5c77736622dcc" exitCode=0
Aug 04 18:53:47.131558 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:47.131134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" event={"ID":"77e4ca22-8569-4063-aa66-e6dd914164fe","Type":"ContainerDied","Data":"743e66d50c1c2700b3c3310aabe96bb5338f7f01ca064c17d7f5c77736622dcc"}
Aug 04 18:53:48.138219 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:48.138180    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" event={"ID":"77e4ca22-8569-4063-aa66-e6dd914164fe","Type":"ContainerStarted","Data":"ba2cce9a8f578f9eb24d16e92aef867e1e4aa850b309f0130fbd0d48e27e1f7e"}
Aug 04 18:53:48.138710 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:48.138266    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" event={"ID":"77e4ca22-8569-4063-aa66-e6dd914164fe","Type":"ContainerStarted","Data":"63c2721c14d6b90987e4ff4d641d1015609a785122544bb00743850b258f0d52"}
Aug 04 18:53:48.138710 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:48.138556    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:48.170566 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:48.170520    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" podStartSLOduration=2.26426618 podStartE2EDuration="6.170504038s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.906237858s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:53:48.168074352 +0000 UTC m=+3227.458688523" watchObservedRunningTime="2026-08-04 18:53:48.170504038 +0000 UTC m=+3227.461118230"
Aug 04 18:53:49.143170 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:49.143144    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:49.144381 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:49.144324    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Aug 04 18:53:50.147143 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:50.147109    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Aug 04 18:53:55.154105 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:55.154021    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:53:55.154727 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:53:55.154698    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Aug 04 18:54:05.154911 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:54:05.154865    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Aug 04 18:54:15.155245 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:54:15.155203    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Aug 04 18:54:25.155193 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:54:25.155152    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Aug 04 18:54:35.155307 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:54:35.155256    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Aug 04 18:54:45.154741 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:54:45.154700    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Aug 04 18:54:55.154595 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:54:55.154552    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Aug 04 18:55:05.154624 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:05.154581    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Aug 04 18:55:10.190462 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:10.190436    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:55:13.251435 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:13.251398    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"]
Aug 04 18:55:13.251876 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:13.251680    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kserve-container" containerID="cri-o://63c2721c14d6b90987e4ff4d641d1015609a785122544bb00743850b258f0d52" gracePeriod=30
Aug 04 18:55:13.251876 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:13.251680    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kube-rbac-proxy" containerID="cri-o://ba2cce9a8f578f9eb24d16e92aef867e1e4aa850b309f0130fbd0d48e27e1f7e" gracePeriod=30
Aug 04 18:55:13.523211 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:13.523119    2566 generic.go:356] "Generic (PLEG): container finished" podID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerID="ba2cce9a8f578f9eb24d16e92aef867e1e4aa850b309f0130fbd0d48e27e1f7e" exitCode=2
Aug 04 18:55:13.523211 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:13.523181    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" event={"ID":"77e4ca22-8569-4063-aa66-e6dd914164fe","Type":"ContainerDied","Data":"ba2cce9a8f578f9eb24d16e92aef867e1e4aa850b309f0130fbd0d48e27e1f7e"}
Aug 04 18:55:15.147568 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:15.147523    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.70:8643/healthz\": dial tcp 10.134.0.70:8643: connect: connection refused"
Aug 04 18:55:17.226725 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.226689    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:55:17.335951 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.335921    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77e4ca22-8569-4063-aa66-e6dd914164fe-cabundle-cert\") pod \"77e4ca22-8569-4063-aa66-e6dd914164fe\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") "
Aug 04 18:55:17.336148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.335969    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwb8h\" (UniqueName: \"kubernetes.io/projected/77e4ca22-8569-4063-aa66-e6dd914164fe-kube-api-access-kwb8h\") pod \"77e4ca22-8569-4063-aa66-e6dd914164fe\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") "
Aug 04 18:55:17.336148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.335992    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77e4ca22-8569-4063-aa66-e6dd914164fe-kube-rbac-proxy-sar-config\") pod \"77e4ca22-8569-4063-aa66-e6dd914164fe\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") "
Aug 04 18:55:17.336148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.336013    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77e4ca22-8569-4063-aa66-e6dd914164fe-kserve-provision-location\") pod \"77e4ca22-8569-4063-aa66-e6dd914164fe\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") "
Aug 04 18:55:17.336148 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.336047    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77e4ca22-8569-4063-aa66-e6dd914164fe-proxy-tls\") pod \"77e4ca22-8569-4063-aa66-e6dd914164fe\" (UID: \"77e4ca22-8569-4063-aa66-e6dd914164fe\") "
Aug 04 18:55:17.336386 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.336335    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e4ca22-8569-4063-aa66-e6dd914164fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "77e4ca22-8569-4063-aa66-e6dd914164fe" (UID: "77e4ca22-8569-4063-aa66-e6dd914164fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:55:17.336386 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.336369    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e4ca22-8569-4063-aa66-e6dd914164fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "77e4ca22-8569-4063-aa66-e6dd914164fe" (UID: "77e4ca22-8569-4063-aa66-e6dd914164fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:55:17.336489 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.336383    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e4ca22-8569-4063-aa66-e6dd914164fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "77e4ca22-8569-4063-aa66-e6dd914164fe" (UID: "77e4ca22-8569-4063-aa66-e6dd914164fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:55:17.338209 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.338168    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e4ca22-8569-4063-aa66-e6dd914164fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77e4ca22-8569-4063-aa66-e6dd914164fe" (UID: "77e4ca22-8569-4063-aa66-e6dd914164fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:55:17.338336 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.338212    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e4ca22-8569-4063-aa66-e6dd914164fe-kube-api-access-kwb8h" (OuterVolumeSpecName: "kube-api-access-kwb8h") pod "77e4ca22-8569-4063-aa66-e6dd914164fe" (UID: "77e4ca22-8569-4063-aa66-e6dd914164fe"). InnerVolumeSpecName "kube-api-access-kwb8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:55:17.437931 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.437837    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwb8h\" (UniqueName: \"kubernetes.io/projected/77e4ca22-8569-4063-aa66-e6dd914164fe-kube-api-access-kwb8h\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:55:17.437931 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.437873    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77e4ca22-8569-4063-aa66-e6dd914164fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:55:17.437931 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.437886    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77e4ca22-8569-4063-aa66-e6dd914164fe-kserve-provision-location\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:55:17.437931 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.437900    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77e4ca22-8569-4063-aa66-e6dd914164fe-proxy-tls\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:55:17.437931 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.437915    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77e4ca22-8569-4063-aa66-e6dd914164fe-cabundle-cert\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:55:17.542049 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.542013    2566 generic.go:356] "Generic (PLEG): container finished" podID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerID="63c2721c14d6b90987e4ff4d641d1015609a785122544bb00743850b258f0d52" exitCode=0
Aug 04 18:55:17.542203 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.542072    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"
Aug 04 18:55:17.542203 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.542079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" event={"ID":"77e4ca22-8569-4063-aa66-e6dd914164fe","Type":"ContainerDied","Data":"63c2721c14d6b90987e4ff4d641d1015609a785122544bb00743850b258f0d52"}
Aug 04 18:55:17.542203 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.542130    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw" event={"ID":"77e4ca22-8569-4063-aa66-e6dd914164fe","Type":"ContainerDied","Data":"fe61695b530bbc9bc750e9efcb9be28cf9a6de3c9f9cb8f42506e4699d5c6515"}
Aug 04 18:55:17.542203 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.542158    2566 scope.go:117] "RemoveContainer" containerID="ba2cce9a8f578f9eb24d16e92aef867e1e4aa850b309f0130fbd0d48e27e1f7e"
Aug 04 18:55:17.553342 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.553316    2566 scope.go:117] "RemoveContainer" containerID="63c2721c14d6b90987e4ff4d641d1015609a785122544bb00743850b258f0d52"
Aug 04 18:55:17.561178 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.561157    2566 scope.go:117] "RemoveContainer" containerID="743e66d50c1c2700b3c3310aabe96bb5338f7f01ca064c17d7f5c77736622dcc"
Aug 04 18:55:17.568643 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.568623    2566 scope.go:117] "RemoveContainer" containerID="ba2cce9a8f578f9eb24d16e92aef867e1e4aa850b309f0130fbd0d48e27e1f7e"
Aug 04 18:55:17.568913 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:55:17.568893    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2cce9a8f578f9eb24d16e92aef867e1e4aa850b309f0130fbd0d48e27e1f7e\": container with ID starting with ba2cce9a8f578f9eb24d16e92aef867e1e4aa850b309f0130fbd0d48e27e1f7e not found: ID does not exist" containerID="ba2cce9a8f578f9eb24d16e92aef867e1e4aa850b309f0130fbd0d48e27e1f7e"
Aug 04 18:55:17.568960 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.568922    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2cce9a8f578f9eb24d16e92aef867e1e4aa850b309f0130fbd0d48e27e1f7e"} err="failed to get container status \"ba2cce9a8f578f9eb24d16e92aef867e1e4aa850b309f0130fbd0d48e27e1f7e\": rpc error: code = NotFound desc = could not find container \"ba2cce9a8f578f9eb24d16e92aef867e1e4aa850b309f0130fbd0d48e27e1f7e\": container with ID starting with ba2cce9a8f578f9eb24d16e92aef867e1e4aa850b309f0130fbd0d48e27e1f7e not found: ID does not exist"
Aug 04 18:55:17.568960 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.568940    2566 scope.go:117] "RemoveContainer" containerID="63c2721c14d6b90987e4ff4d641d1015609a785122544bb00743850b258f0d52"
Aug 04 18:55:17.569181 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:55:17.569164    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c2721c14d6b90987e4ff4d641d1015609a785122544bb00743850b258f0d52\": container with ID starting with 63c2721c14d6b90987e4ff4d641d1015609a785122544bb00743850b258f0d52 not found: ID does not exist" containerID="63c2721c14d6b90987e4ff4d641d1015609a785122544bb00743850b258f0d52"
Aug 04 18:55:17.569220 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.569187    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c2721c14d6b90987e4ff4d641d1015609a785122544bb00743850b258f0d52"} err="failed to get container status \"63c2721c14d6b90987e4ff4d641d1015609a785122544bb00743850b258f0d52\": rpc error: code = NotFound desc = could not find container \"63c2721c14d6b90987e4ff4d641d1015609a785122544bb00743850b258f0d52\": container with ID starting with 63c2721c14d6b90987e4ff4d641d1015609a785122544bb00743850b258f0d52 not found: ID does not exist"
Aug 04 18:55:17.569220 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.569202    2566 scope.go:117] "RemoveContainer" containerID="743e66d50c1c2700b3c3310aabe96bb5338f7f01ca064c17d7f5c77736622dcc"
Aug 04 18:55:17.569459 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:55:17.569443    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743e66d50c1c2700b3c3310aabe96bb5338f7f01ca064c17d7f5c77736622dcc\": container with ID starting with 743e66d50c1c2700b3c3310aabe96bb5338f7f01ca064c17d7f5c77736622dcc not found: ID does not exist" containerID="743e66d50c1c2700b3c3310aabe96bb5338f7f01ca064c17d7f5c77736622dcc"
Aug 04 18:55:17.569511 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.569463    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743e66d50c1c2700b3c3310aabe96bb5338f7f01ca064c17d7f5c77736622dcc"} err="failed to get container status \"743e66d50c1c2700b3c3310aabe96bb5338f7f01ca064c17d7f5c77736622dcc\": rpc error: code = NotFound desc = could not find container \"743e66d50c1c2700b3c3310aabe96bb5338f7f01ca064c17d7f5c77736622dcc\": container with ID starting with 743e66d50c1c2700b3c3310aabe96bb5338f7f01ca064c17d7f5c77736622dcc not found: ID does not exist"
Aug 04 18:55:17.576572 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.576546    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"]
Aug 04 18:55:17.586256 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:17.586236    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9krkw"]
Aug 04 18:55:19.190553 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.190511    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-89t52/must-gather-9j4g5"]
Aug 04 18:55:19.191283 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.191260    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="storage-initializer"
Aug 04 18:55:19.191418 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.191287    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="storage-initializer"
Aug 04 18:55:19.191418 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.191331    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kserve-container"
Aug 04 18:55:19.191418 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.191340    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kserve-container"
Aug 04 18:55:19.191418 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.191379    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kube-rbac-proxy"
Aug 04 18:55:19.191418 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.191389    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kube-rbac-proxy"
Aug 04 18:55:19.191668 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.191570    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kube-rbac-proxy"
Aug 04 18:55:19.191668 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.191590    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" containerName="kserve-container"
Aug 04 18:55:19.197149 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.197117    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-89t52/must-gather-9j4g5"
Aug 04 18:55:19.200055 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.200036    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-89t52\"/\"openshift-service-ca.crt\""
Aug 04 18:55:19.200213 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.200036    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-89t52\"/\"default-dockercfg-qkcg4\""
Aug 04 18:55:19.200213 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.200140    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-89t52\"/\"kube-root-ca.crt\""
Aug 04 18:55:19.203202 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.203175    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77e4ca22-8569-4063-aa66-e6dd914164fe" path="/var/lib/kubelet/pods/77e4ca22-8569-4063-aa66-e6dd914164fe/volumes"
Aug 04 18:55:19.211153 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.211131    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-89t52/must-gather-9j4g5"]
Aug 04 18:55:19.355349 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.355313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8fpt\" (UniqueName: \"kubernetes.io/projected/32a015cb-c9e4-4478-afd3-02588c319b5c-kube-api-access-w8fpt\") pod \"must-gather-9j4g5\" (UID: \"32a015cb-c9e4-4478-afd3-02588c319b5c\") " pod="openshift-must-gather-89t52/must-gather-9j4g5"
Aug 04 18:55:19.355567 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.355447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/32a015cb-c9e4-4478-afd3-02588c319b5c-must-gather-output\") pod \"must-gather-9j4g5\" (UID: \"32a015cb-c9e4-4478-afd3-02588c319b5c\") " pod="openshift-must-gather-89t52/must-gather-9j4g5"
Aug 04 18:55:19.456269 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.456171    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/32a015cb-c9e4-4478-afd3-02588c319b5c-must-gather-output\") pod \"must-gather-9j4g5\" (UID: \"32a015cb-c9e4-4478-afd3-02588c319b5c\") " pod="openshift-must-gather-89t52/must-gather-9j4g5"
Aug 04 18:55:19.456269 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.456234    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8fpt\" (UniqueName: \"kubernetes.io/projected/32a015cb-c9e4-4478-afd3-02588c319b5c-kube-api-access-w8fpt\") pod \"must-gather-9j4g5\" (UID: \"32a015cb-c9e4-4478-afd3-02588c319b5c\") " pod="openshift-must-gather-89t52/must-gather-9j4g5"
Aug 04 18:55:19.456617 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.456595    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/32a015cb-c9e4-4478-afd3-02588c319b5c-must-gather-output\") pod \"must-gather-9j4g5\" (UID: \"32a015cb-c9e4-4478-afd3-02588c319b5c\") " pod="openshift-must-gather-89t52/must-gather-9j4g5"
Aug 04 18:55:19.468496 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.468468    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8fpt\" (UniqueName: \"kubernetes.io/projected/32a015cb-c9e4-4478-afd3-02588c319b5c-kube-api-access-w8fpt\") pod \"must-gather-9j4g5\" (UID: \"32a015cb-c9e4-4478-afd3-02588c319b5c\") " pod="openshift-must-gather-89t52/must-gather-9j4g5"
Aug 04 18:55:19.516994 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.516958    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-89t52/must-gather-9j4g5"
Aug 04 18:55:19.656400 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.656372    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-89t52/must-gather-9j4g5"]
Aug 04 18:55:19.658390 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:55:19.658362    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32a015cb_c9e4_4478_afd3_02588c319b5c.slice/crio-18d46cd10c21453bb93a8efe7e8e4deacb732d3dd60ff1d2c470a3f48545ae1b WatchSource:0}: Error finding container 18d46cd10c21453bb93a8efe7e8e4deacb732d3dd60ff1d2c470a3f48545ae1b: Status 404 returned error can't find the container with id 18d46cd10c21453bb93a8efe7e8e4deacb732d3dd60ff1d2c470a3f48545ae1b
Aug 04 18:55:19.660144 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:19.660128    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:55:20.557413 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:20.557379    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-89t52/must-gather-9j4g5" event={"ID":"32a015cb-c9e4-4478-afd3-02588c319b5c","Type":"ContainerStarted","Data":"18d46cd10c21453bb93a8efe7e8e4deacb732d3dd60ff1d2c470a3f48545ae1b"}
Aug 04 18:55:27.599963 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:27.599920    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-89t52/must-gather-9j4g5" event={"ID":"32a015cb-c9e4-4478-afd3-02588c319b5c","Type":"ContainerStarted","Data":"b50fd5251ce246fa6859b90170f2c3574c856f337914ec436481e33d78dfeb8a"}
Aug 04 18:55:27.600399 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:27.599982    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-89t52/must-gather-9j4g5" event={"ID":"32a015cb-c9e4-4478-afd3-02588c319b5c","Type":"ContainerStarted","Data":"c2a0cc5f8816d15c94790837faa61a36638fc6f718794bbeb69740fb4e897e5e"}
Aug 04 18:55:27.624718 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:27.624665    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-89t52/must-gather-9j4g5" podStartSLOduration=1.401333219 podStartE2EDuration="8.624648312s" totalImagesPullingTime="7.223315093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:55:27.624026308 +0000 UTC m=+3326.914640483" watchObservedRunningTime="2026-08-04 18:55:27.624648312 +0000 UTC m=+3326.915262482"
Aug 04 18:55:55.731064 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:55.731034    2566 generic.go:356] "Generic (PLEG): container finished" podID="32a015cb-c9e4-4478-afd3-02588c319b5c" containerID="c2a0cc5f8816d15c94790837faa61a36638fc6f718794bbeb69740fb4e897e5e" exitCode=0
Aug 04 18:55:55.731502 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:55.731107    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-89t52/must-gather-9j4g5" event={"ID":"32a015cb-c9e4-4478-afd3-02588c319b5c","Type":"ContainerDied","Data":"c2a0cc5f8816d15c94790837faa61a36638fc6f718794bbeb69740fb4e897e5e"}
Aug 04 18:55:55.731502 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:55.731477    2566 scope.go:117] "RemoveContainer" containerID="c2a0cc5f8816d15c94790837faa61a36638fc6f718794bbeb69740fb4e897e5e"
Aug 04 18:55:55.776944 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:55.776915    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-89t52_must-gather-9j4g5_32a015cb-c9e4-4478-afd3-02588c319b5c/gather/0.log"
Aug 04 18:55:56.540554 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:56.540523    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vp9hh/must-gather-7rw8j"]
Aug 04 18:55:56.544071 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:56.544054    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vp9hh/must-gather-7rw8j"
Aug 04 18:55:56.546863 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:56.546841    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vp9hh\"/\"openshift-service-ca.crt\""
Aug 04 18:55:56.546980 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:56.546891    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vp9hh\"/\"kube-root-ca.crt\""
Aug 04 18:55:56.546980 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:56.546910    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vp9hh\"/\"default-dockercfg-vqdkp\""
Aug 04 18:55:56.560753 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:56.560734    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vp9hh/must-gather-7rw8j"]
Aug 04 18:55:56.679690 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:56.679660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kknjq\" (UniqueName: \"kubernetes.io/projected/c880e9fc-e2bd-4b6c-9458-5f5e51573bbf-kube-api-access-kknjq\") pod \"must-gather-7rw8j\" (UID: \"c880e9fc-e2bd-4b6c-9458-5f5e51573bbf\") " pod="openshift-must-gather-vp9hh/must-gather-7rw8j"
Aug 04 18:55:56.679874 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:56.679728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c880e9fc-e2bd-4b6c-9458-5f5e51573bbf-must-gather-output\") pod \"must-gather-7rw8j\" (UID: \"c880e9fc-e2bd-4b6c-9458-5f5e51573bbf\") " pod="openshift-must-gather-vp9hh/must-gather-7rw8j"
Aug 04 18:55:56.780905 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:56.780872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c880e9fc-e2bd-4b6c-9458-5f5e51573bbf-must-gather-output\") pod \"must-gather-7rw8j\" (UID: \"c880e9fc-e2bd-4b6c-9458-5f5e51573bbf\") " pod="openshift-must-gather-vp9hh/must-gather-7rw8j"
Aug 04 18:55:56.781391 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:56.780942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kknjq\" (UniqueName: \"kubernetes.io/projected/c880e9fc-e2bd-4b6c-9458-5f5e51573bbf-kube-api-access-kknjq\") pod \"must-gather-7rw8j\" (UID: \"c880e9fc-e2bd-4b6c-9458-5f5e51573bbf\") " pod="openshift-must-gather-vp9hh/must-gather-7rw8j"
Aug 04 18:55:56.781391 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:56.781283    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c880e9fc-e2bd-4b6c-9458-5f5e51573bbf-must-gather-output\") pod \"must-gather-7rw8j\" (UID: \"c880e9fc-e2bd-4b6c-9458-5f5e51573bbf\") " pod="openshift-must-gather-vp9hh/must-gather-7rw8j"
Aug 04 18:55:56.793135 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:56.793076    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kknjq\" (UniqueName: \"kubernetes.io/projected/c880e9fc-e2bd-4b6c-9458-5f5e51573bbf-kube-api-access-kknjq\") pod \"must-gather-7rw8j\" (UID: \"c880e9fc-e2bd-4b6c-9458-5f5e51573bbf\") " pod="openshift-must-gather-vp9hh/must-gather-7rw8j"
Aug 04 18:55:56.852536 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:56.852505    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vp9hh/must-gather-7rw8j"
Aug 04 18:55:56.994635 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:56.994610    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vp9hh/must-gather-7rw8j"]
Aug 04 18:55:56.997165 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:55:56.997136    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc880e9fc_e2bd_4b6c_9458_5f5e51573bbf.slice/crio-46daa3065290f51fa62f9a71aef4f145a7a72c2e081e2c1a11091f3d05b0517c WatchSource:0}: Error finding container 46daa3065290f51fa62f9a71aef4f145a7a72c2e081e2c1a11091f3d05b0517c: Status 404 returned error can't find the container with id 46daa3065290f51fa62f9a71aef4f145a7a72c2e081e2c1a11091f3d05b0517c
Aug 04 18:55:57.739940 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:57.739907    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vp9hh/must-gather-7rw8j" event={"ID":"c880e9fc-e2bd-4b6c-9458-5f5e51573bbf","Type":"ContainerStarted","Data":"46daa3065290f51fa62f9a71aef4f145a7a72c2e081e2c1a11091f3d05b0517c"}
Aug 04 18:55:59.751579 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:59.751493    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vp9hh/must-gather-7rw8j" event={"ID":"c880e9fc-e2bd-4b6c-9458-5f5e51573bbf","Type":"ContainerStarted","Data":"a0b1e2e83706abbc5c485a4a42bc0c934860768ddbe1809e5a0f761d7d6e95f7"}
Aug 04 18:55:59.751579 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:59.751544    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vp9hh/must-gather-7rw8j" event={"ID":"c880e9fc-e2bd-4b6c-9458-5f5e51573bbf","Type":"ContainerStarted","Data":"f1394099b3adea69ef9a08eddebeac93658001b5a18ee4292a9ab4965fd49b18"}
Aug 04 18:55:59.778573 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:55:59.778503    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vp9hh/must-gather-7rw8j" podStartSLOduration=1.4645807 podStartE2EDuration="3.778481856s" totalImagesPullingTime="2.313901156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:55:59.777161744 +0000 UTC m=+3359.067775917" watchObservedRunningTime="2026-08-04 18:55:59.778481856 +0000 UTC m=+3359.069096027"
Aug 04 18:56:01.399366 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.399323    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-89t52/must-gather-9j4g5"]
Aug 04 18:56:01.399827 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.399529    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-89t52/must-gather-9j4g5" podUID="32a015cb-c9e4-4478-afd3-02588c319b5c" containerName="copy" containerID="cri-o://b50fd5251ce246fa6859b90170f2c3574c856f337914ec436481e33d78dfeb8a" gracePeriod=2
Aug 04 18:56:01.404909 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.404609    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-89t52/must-gather-9j4g5"]
Aug 04 18:56:01.417738 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:56:01.417699    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9j4g5\" is forbidden: User \"system:node:ip-10-0-132-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-89t52\": no relationship found between node 'ip-10-0-132-242.ec2.internal' and this object" podUID="32a015cb-c9e4-4478-afd3-02588c319b5c" pod="openshift-must-gather-89t52/must-gather-9j4g5"
Aug 04 18:56:01.534787 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.534743    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t46pw_6cbc3f49-e03c-425d-9da2-71eb15915d42/global-pull-secret-syncer/0.log"
Aug 04 18:56:01.647187 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.647159    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-89t52_must-gather-9j4g5_32a015cb-c9e4-4478-afd3-02588c319b5c/copy/0.log"
Aug 04 18:56:01.653032 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.652531    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-89t52/must-gather-9j4g5"
Aug 04 18:56:01.657314 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:56:01.657282    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9j4g5\" is forbidden: User \"system:node:ip-10-0-132-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-89t52\": no relationship found between node 'ip-10-0-132-242.ec2.internal' and this object" podUID="32a015cb-c9e4-4478-afd3-02588c319b5c" pod="openshift-must-gather-89t52/must-gather-9j4g5"
Aug 04 18:56:01.720737 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.720701    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wjjnh_54a81028-407e-4ddd-b7bd-9698d6e35f2e/konnectivity-agent/0.log"
Aug 04 18:56:01.730298 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.730270    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/32a015cb-c9e4-4478-afd3-02588c319b5c-must-gather-output\") pod \"32a015cb-c9e4-4478-afd3-02588c319b5c\" (UID: \"32a015cb-c9e4-4478-afd3-02588c319b5c\") "
Aug 04 18:56:01.730505 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.730385    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8fpt\" (UniqueName: \"kubernetes.io/projected/32a015cb-c9e4-4478-afd3-02588c319b5c-kube-api-access-w8fpt\") pod \"32a015cb-c9e4-4478-afd3-02588c319b5c\" (UID: \"32a015cb-c9e4-4478-afd3-02588c319b5c\") "
Aug 04 18:56:01.731941 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.731905    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a015cb-c9e4-4478-afd3-02588c319b5c-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "32a015cb-c9e4-4478-afd3-02588c319b5c" (UID: "32a015cb-c9e4-4478-afd3-02588c319b5c"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:56:01.732822 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.732796    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a015cb-c9e4-4478-afd3-02588c319b5c-kube-api-access-w8fpt" (OuterVolumeSpecName: "kube-api-access-w8fpt") pod "32a015cb-c9e4-4478-afd3-02588c319b5c" (UID: "32a015cb-c9e4-4478-afd3-02588c319b5c"). InnerVolumeSpecName "kube-api-access-w8fpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:56:01.764240 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.764215    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-89t52_must-gather-9j4g5_32a015cb-c9e4-4478-afd3-02588c319b5c/copy/0.log"
Aug 04 18:56:01.764695 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.764665    2566 generic.go:356] "Generic (PLEG): container finished" podID="32a015cb-c9e4-4478-afd3-02588c319b5c" containerID="b50fd5251ce246fa6859b90170f2c3574c856f337914ec436481e33d78dfeb8a" exitCode=143
Aug 04 18:56:01.764863 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.764724    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-89t52/must-gather-9j4g5"
Aug 04 18:56:01.764863 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.764726    2566 scope.go:117] "RemoveContainer" containerID="b50fd5251ce246fa6859b90170f2c3574c856f337914ec436481e33d78dfeb8a"
Aug 04 18:56:01.767660 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:56:01.767627    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9j4g5\" is forbidden: User \"system:node:ip-10-0-132-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-89t52\": no relationship found between node 'ip-10-0-132-242.ec2.internal' and this object" podUID="32a015cb-c9e4-4478-afd3-02588c319b5c" pod="openshift-must-gather-89t52/must-gather-9j4g5"
Aug 04 18:56:01.775198 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.775179    2566 scope.go:117] "RemoveContainer" containerID="c2a0cc5f8816d15c94790837faa61a36638fc6f718794bbeb69740fb4e897e5e"
Aug 04 18:56:01.776632 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:56:01.776600    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9j4g5\" is forbidden: User \"system:node:ip-10-0-132-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-89t52\": no relationship found between node 'ip-10-0-132-242.ec2.internal' and this object" podUID="32a015cb-c9e4-4478-afd3-02588c319b5c" pod="openshift-must-gather-89t52/must-gather-9j4g5"
Aug 04 18:56:01.787673 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.787650    2566 scope.go:117] "RemoveContainer" containerID="b50fd5251ce246fa6859b90170f2c3574c856f337914ec436481e33d78dfeb8a"
Aug 04 18:56:01.788003 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:56:01.787982    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b50fd5251ce246fa6859b90170f2c3574c856f337914ec436481e33d78dfeb8a\": container with ID starting with b50fd5251ce246fa6859b90170f2c3574c856f337914ec436481e33d78dfeb8a not found: ID does not exist" containerID="b50fd5251ce246fa6859b90170f2c3574c856f337914ec436481e33d78dfeb8a"
Aug 04 18:56:01.788097 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.788014    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b50fd5251ce246fa6859b90170f2c3574c856f337914ec436481e33d78dfeb8a"} err="failed to get container status \"b50fd5251ce246fa6859b90170f2c3574c856f337914ec436481e33d78dfeb8a\": rpc error: code = NotFound desc = could not find container \"b50fd5251ce246fa6859b90170f2c3574c856f337914ec436481e33d78dfeb8a\": container with ID starting with b50fd5251ce246fa6859b90170f2c3574c856f337914ec436481e33d78dfeb8a not found: ID does not exist"
Aug 04 18:56:01.788097 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.788034    2566 scope.go:117] "RemoveContainer" containerID="c2a0cc5f8816d15c94790837faa61a36638fc6f718794bbeb69740fb4e897e5e"
Aug 04 18:56:01.788321 ip-10-0-132-242 kubenswrapper[2566]: E0804 18:56:01.788292    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a0cc5f8816d15c94790837faa61a36638fc6f718794bbeb69740fb4e897e5e\": container with ID starting with c2a0cc5f8816d15c94790837faa61a36638fc6f718794bbeb69740fb4e897e5e not found: ID does not exist" containerID="c2a0cc5f8816d15c94790837faa61a36638fc6f718794bbeb69740fb4e897e5e"
Aug 04 18:56:01.788401 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.788330    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a0cc5f8816d15c94790837faa61a36638fc6f718794bbeb69740fb4e897e5e"} err="failed to get container status \"c2a0cc5f8816d15c94790837faa61a36638fc6f718794bbeb69740fb4e897e5e\": rpc error: code = NotFound desc = could not find container \"c2a0cc5f8816d15c94790837faa61a36638fc6f718794bbeb69740fb4e897e5e\": container with ID starting with c2a0cc5f8816d15c94790837faa61a36638fc6f718794bbeb69740fb4e897e5e not found: ID does not exist"
Aug 04 18:56:01.831421 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.831391    2566 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/32a015cb-c9e4-4478-afd3-02588c319b5c-must-gather-output\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:56:01.831421 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.831417    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8fpt\" (UniqueName: \"kubernetes.io/projected/32a015cb-c9e4-4478-afd3-02588c319b5c-kube-api-access-w8fpt\") on node \"ip-10-0-132-242.ec2.internal\" DevicePath \"\""
Aug 04 18:56:01.850168 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:01.850135    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-242.ec2.internal_4b78a122ed0f6c6c547f0087e492e65b/haproxy/0.log"
Aug 04 18:56:03.199449 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:03.199408    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32a015cb-c9e4-4478-afd3-02588c319b5c" path="/var/lib/kubelet/pods/32a015cb-c9e4-4478-afd3-02588c319b5c/volumes"
Aug 04 18:56:05.411716 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:05.411665    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_82af722f-f621-4ca4-9586-c6fbf84591b2/alertmanager/0.log"
Aug 04 18:56:05.466610 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:05.466573    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_82af722f-f621-4ca4-9586-c6fbf84591b2/config-reloader/0.log"
Aug 04 18:56:05.523994 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:05.523966    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_82af722f-f621-4ca4-9586-c6fbf84591b2/kube-rbac-proxy-web/0.log"
Aug 04 18:56:05.569688 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:05.569660    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_82af722f-f621-4ca4-9586-c6fbf84591b2/kube-rbac-proxy/0.log"
Aug 04 18:56:05.620881 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:05.620803    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_82af722f-f621-4ca4-9586-c6fbf84591b2/kube-rbac-proxy-metric/0.log"
Aug 04 18:56:05.667209 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:05.667136    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_82af722f-f621-4ca4-9586-c6fbf84591b2/prom-label-proxy/0.log"
Aug 04 18:56:05.720744 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:05.720714    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_82af722f-f621-4ca4-9586-c6fbf84591b2/init-config-reloader/0.log"
Aug 04 18:56:05.898823 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:05.898790    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-h46zx_de118084-296a-4dfb-b5ed-4f869a8e1b50/kube-state-metrics/0.log"
Aug 04 18:56:05.959259 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:05.959174    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-h46zx_de118084-296a-4dfb-b5ed-4f869a8e1b50/kube-rbac-proxy-main/0.log"
Aug 04 18:56:06.029304 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:06.029268    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-h46zx_de118084-296a-4dfb-b5ed-4f869a8e1b50/kube-rbac-proxy-self/0.log"
Aug 04 18:56:06.101113 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:06.101077    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-657674cc-656rm_2664c33b-5edb-4026-adb7-ca112956f898/metrics-server/0.log"
Aug 04 18:56:06.172156 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:06.172123    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-4zk4t_ec4c43a8-531d-4176-a3a7-51541eb13a3e/monitoring-plugin/0.log"
Aug 04 18:56:06.381770 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:06.381745    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k5tqn_64a8adba-c9b7-4a79-826a-d64a1668f4cf/node-exporter/0.log"
Aug 04 18:56:06.438431 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:06.438403    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k5tqn_64a8adba-c9b7-4a79-826a-d64a1668f4cf/kube-rbac-proxy/0.log"
Aug 04 18:56:06.487157 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:06.487074    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k5tqn_64a8adba-c9b7-4a79-826a-d64a1668f4cf/init-textfile/0.log"
Aug 04 18:56:06.627197 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:06.627170    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-kh899_d0f3e4e0-1cb2-4782-a160-b2d4c530fc48/kube-rbac-proxy-main/0.log"
Aug 04 18:56:06.667212 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:06.667179    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-kh899_d0f3e4e0-1cb2-4782-a160-b2d4c530fc48/kube-rbac-proxy-self/0.log"
Aug 04 18:56:06.711995 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:06.711955    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-kh899_d0f3e4e0-1cb2-4782-a160-b2d4c530fc48/openshift-state-metrics/0.log"
Aug 04 18:56:06.792222 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:06.792135    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_af261892-1e1c-4d98-9aaf-5a292f5c78f5/prometheus/0.log"
Aug 04 18:56:06.847162 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:06.847137    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_af261892-1e1c-4d98-9aaf-5a292f5c78f5/config-reloader/0.log"
Aug 04 18:56:06.894641 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:06.894602    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_af261892-1e1c-4d98-9aaf-5a292f5c78f5/thanos-sidecar/0.log"
Aug 04 18:56:06.953589 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:06.953553    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_af261892-1e1c-4d98-9aaf-5a292f5c78f5/kube-rbac-proxy-web/0.log"
Aug 04 18:56:06.998896 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:06.998863    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_af261892-1e1c-4d98-9aaf-5a292f5c78f5/kube-rbac-proxy/0.log"
Aug 04 18:56:07.046720 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:07.046648    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_af261892-1e1c-4d98-9aaf-5a292f5c78f5/kube-rbac-proxy-thanos/0.log"
Aug 04 18:56:07.107332 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:07.107299    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_af261892-1e1c-4d98-9aaf-5a292f5c78f5/init-config-reloader/0.log"
Aug 04 18:56:07.161049 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:07.161020    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-xb49f_45ea4544-fff5-40fd-9fa3-60fc77126170/prometheus-operator/0.log"
Aug 04 18:56:07.217561 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:07.217531    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-xb49f_45ea4544-fff5-40fd-9fa3-60fc77126170/kube-rbac-proxy/0.log"
Aug 04 18:56:07.267238 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:07.267207    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-vrdvt_49a382ef-5436-4392-a0ac-327454be1c2d/prometheus-operator-admission-webhook/0.log"
Aug 04 18:56:07.319868 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:07.319824    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84f56b86-95mmb_f36fbfde-f663-49b3-b8c1-6160b8243eca/telemeter-client/0.log"
Aug 04 18:56:07.366456 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:07.366426    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84f56b86-95mmb_f36fbfde-f663-49b3-b8c1-6160b8243eca/reload/0.log"
Aug 04 18:56:07.413243 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:07.413209    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84f56b86-95mmb_f36fbfde-f663-49b3-b8c1-6160b8243eca/kube-rbac-proxy/0.log"
Aug 04 18:56:07.466650 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:07.466620    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7588cc9ffd-r4r52_659adef3-366c-4c1d-b634-650ff98fbb02/thanos-query/0.log"
Aug 04 18:56:07.510694 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:07.510663    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7588cc9ffd-r4r52_659adef3-366c-4c1d-b634-650ff98fbb02/kube-rbac-proxy-web/0.log"
Aug 04 18:56:07.555177 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:07.555147    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7588cc9ffd-r4r52_659adef3-366c-4c1d-b634-650ff98fbb02/kube-rbac-proxy/0.log"
Aug 04 18:56:07.600526 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:07.600347    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7588cc9ffd-r4r52_659adef3-366c-4c1d-b634-650ff98fbb02/prom-label-proxy/0.log"
Aug 04 18:56:07.637703 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:07.637672    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7588cc9ffd-r4r52_659adef3-366c-4c1d-b634-650ff98fbb02/kube-rbac-proxy-rules/0.log"
Aug 04 18:56:07.688227 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:07.688196    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7588cc9ffd-r4r52_659adef3-366c-4c1d-b634-650ff98fbb02/kube-rbac-proxy-metrics/0.log"
Aug 04 18:56:09.574953 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.574916    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"]
Aug 04 18:56:09.575548 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.575524    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a015cb-c9e4-4478-afd3-02588c319b5c" containerName="gather"
Aug 04 18:56:09.575548 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.575547    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a015cb-c9e4-4478-afd3-02588c319b5c" containerName="gather"
Aug 04 18:56:09.575746 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.575576    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a015cb-c9e4-4478-afd3-02588c319b5c" containerName="copy"
Aug 04 18:56:09.575746 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.575586    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a015cb-c9e4-4478-afd3-02588c319b5c" containerName="copy"
Aug 04 18:56:09.575746 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.575673    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="32a015cb-c9e4-4478-afd3-02588c319b5c" containerName="copy"
Aug 04 18:56:09.575746 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.575689    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="32a015cb-c9e4-4478-afd3-02588c319b5c" containerName="gather"
Aug 04 18:56:09.581150 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.581125    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:09.589766 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.589167    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"]
Aug 04 18:56:09.717301 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.717267    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfc0af6d-e61c-472c-a865-abca83596b44-lib-modules\") pod \"perf-node-gather-daemonset-9tgzj\" (UID: \"bfc0af6d-e61c-472c-a865-abca83596b44\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:09.717638 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.717612    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfc0af6d-e61c-472c-a865-abca83596b44-proc\") pod \"perf-node-gather-daemonset-9tgzj\" (UID: \"bfc0af6d-e61c-472c-a865-abca83596b44\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:09.717773 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.717757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfc0af6d-e61c-472c-a865-abca83596b44-sys\") pod \"perf-node-gather-daemonset-9tgzj\" (UID: \"bfc0af6d-e61c-472c-a865-abca83596b44\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:09.717936 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.717920    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfc0af6d-e61c-472c-a865-abca83596b44-podres\") pod \"perf-node-gather-daemonset-9tgzj\" (UID: \"bfc0af6d-e61c-472c-a865-abca83596b44\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:09.718063 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.718047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5chw\" (UniqueName: \"kubernetes.io/projected/bfc0af6d-e61c-472c-a865-abca83596b44-kube-api-access-m5chw\") pod \"perf-node-gather-daemonset-9tgzj\" (UID: \"bfc0af6d-e61c-472c-a865-abca83596b44\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:09.819663 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.819628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfc0af6d-e61c-472c-a865-abca83596b44-lib-modules\") pod \"perf-node-gather-daemonset-9tgzj\" (UID: \"bfc0af6d-e61c-472c-a865-abca83596b44\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:09.819866 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.819719    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfc0af6d-e61c-472c-a865-abca83596b44-proc\") pod \"perf-node-gather-daemonset-9tgzj\" (UID: \"bfc0af6d-e61c-472c-a865-abca83596b44\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:09.819866 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.819745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfc0af6d-e61c-472c-a865-abca83596b44-sys\") pod \"perf-node-gather-daemonset-9tgzj\" (UID: \"bfc0af6d-e61c-472c-a865-abca83596b44\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:09.819866 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.819783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfc0af6d-e61c-472c-a865-abca83596b44-podres\") pod \"perf-node-gather-daemonset-9tgzj\" (UID: \"bfc0af6d-e61c-472c-a865-abca83596b44\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:09.819866 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.819808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5chw\" (UniqueName: \"kubernetes.io/projected/bfc0af6d-e61c-472c-a865-abca83596b44-kube-api-access-m5chw\") pod \"perf-node-gather-daemonset-9tgzj\" (UID: \"bfc0af6d-e61c-472c-a865-abca83596b44\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:09.820049 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.819872    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfc0af6d-e61c-472c-a865-abca83596b44-sys\") pod \"perf-node-gather-daemonset-9tgzj\" (UID: \"bfc0af6d-e61c-472c-a865-abca83596b44\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:09.820049 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.819896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfc0af6d-e61c-472c-a865-abca83596b44-proc\") pod \"perf-node-gather-daemonset-9tgzj\" (UID: \"bfc0af6d-e61c-472c-a865-abca83596b44\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:09.820049 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.819842    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfc0af6d-e61c-472c-a865-abca83596b44-lib-modules\") pod \"perf-node-gather-daemonset-9tgzj\" (UID: \"bfc0af6d-e61c-472c-a865-abca83596b44\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:09.820049 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.819952    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfc0af6d-e61c-472c-a865-abca83596b44-podres\") pod \"perf-node-gather-daemonset-9tgzj\" (UID: \"bfc0af6d-e61c-472c-a865-abca83596b44\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:09.830881 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.830812    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5chw\" (UniqueName: \"kubernetes.io/projected/bfc0af6d-e61c-472c-a865-abca83596b44-kube-api-access-m5chw\") pod \"perf-node-gather-daemonset-9tgzj\" (UID: \"bfc0af6d-e61c-472c-a865-abca83596b44\") " pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:09.893733 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:09.893700    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:10.060608 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:10.060573    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"]
Aug 04 18:56:10.061843 ip-10-0-132-242 kubenswrapper[2566]: W0804 18:56:10.061809    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbfc0af6d_e61c_472c_a865_abca83596b44.slice/crio-8256808cb06d2ce9759f7e123b1ee6eb47c6e46863a95a68bf8e88cc7395894a WatchSource:0}: Error finding container 8256808cb06d2ce9759f7e123b1ee6eb47c6e46863a95a68bf8e88cc7395894a: Status 404 returned error can't find the container with id 8256808cb06d2ce9759f7e123b1ee6eb47c6e46863a95a68bf8e88cc7395894a
Aug 04 18:56:10.842610 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:10.842564    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj" event={"ID":"bfc0af6d-e61c-472c-a865-abca83596b44","Type":"ContainerStarted","Data":"d613c17e45cd9f5b00251b8bd0a2e77a9aae3642fa92b56ec3efc5ff2ec0f95c"}
Aug 04 18:56:10.843100 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:10.842635    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj" event={"ID":"bfc0af6d-e61c-472c-a865-abca83596b44","Type":"ContainerStarted","Data":"8256808cb06d2ce9759f7e123b1ee6eb47c6e46863a95a68bf8e88cc7395894a"}
Aug 04 18:56:10.843100 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:10.842664    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:10.864037 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:10.861861    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj" podStartSLOduration=1.861837934 podStartE2EDuration="1.861837934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:56:10.861114331 +0000 UTC m=+3370.151728523" watchObservedRunningTime="2026-08-04 18:56:10.861837934 +0000 UTC m=+3370.152452107"
Aug 04 18:56:11.775738 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:11.775709    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xk4xc_f8f16e23-a10c-4e3b-91e9-e14582c60d00/dns/0.log"
Aug 04 18:56:11.810936 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:11.810909    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xk4xc_f8f16e23-a10c-4e3b-91e9-e14582c60d00/kube-rbac-proxy/0.log"
Aug 04 18:56:11.964198 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:11.964172    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v69qf_703eadd7-667a-4879-aa4e-2ccedb5b7c8a/dns-node-resolver/0.log"
Aug 04 18:56:12.542755 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:12.542730    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7f2bk_7dce6b12-1695-48e9-b13a-0bf64d3107ff/node-ca/0.log"
Aug 04 18:56:13.906454 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:13.906419    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fpdw5_ec4a4cad-5eab-4f95-99ba-a918bfed16c9/serve-healthcheck-canary/0.log"
Aug 04 18:56:14.387686 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:14.387658    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dkvs8_8d9301b5-6d5a-4e67-9bfd-cfc549bedf26/kube-rbac-proxy/0.log"
Aug 04 18:56:14.415290 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:14.415264    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dkvs8_8d9301b5-6d5a-4e67-9bfd-cfc549bedf26/exporter/0.log"
Aug 04 18:56:14.443675 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:14.443612    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dkvs8_8d9301b5-6d5a-4e67-9bfd-cfc549bedf26/extractor/0.log"
Aug 04 18:56:16.857184 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:16.857155    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vp9hh/perf-node-gather-daemonset-9tgzj"
Aug 04 18:56:17.678136 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:17.678100    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-8588677cfb-r2pkl_dc6b2f08-4c9e-44a0-a1da-4e63bfb92dc3/manager/0.log"
Aug 04 18:56:17.735527 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:17.735495    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-rtpqq_93aeb3ba-444e-4165-901a-cd2512c2d6dc/s3-init/0.log"
Aug 04 18:56:17.763614 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:17.763589    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-rtpqq_93aeb3ba-444e-4165-901a-cd2512c2d6dc/download-hf-model/0.log"
Aug 04 18:56:17.806022 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:17.805993    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-s6qrd_52b71d8a-53dd-4b07-9cbd-6b08ee2343c3/s3-tls-init-custom/0.log"
Aug 04 18:56:17.830756 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:17.830721    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-s6qrd_52b71d8a-53dd-4b07-9cbd-6b08ee2343c3/download-hf-model/0.log"
Aug 04 18:56:17.878335 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:17.878307    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-v6pwv_f4a5204e-1056-4c95-8c20-14ecde19ac08/s3-tls-init-serving/0.log"
Aug 04 18:56:17.904269 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:17.904231    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-v6pwv_f4a5204e-1056-4c95-8c20-14ecde19ac08/download-hf-model/0.log"
Aug 04 18:56:17.992913 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:17.992827    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-kkq9j_67d8f728-7658-4e60-852d-c879cf1cc59c/seaweedfs-tls-custom/0.log"
Aug 04 18:56:18.031068 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:18.031038    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-m65d9_f1fb9e9a-79ff-4cb8-a17d-b1d1699be158/seaweedfs-tls-serving/0.log"
Aug 04 18:56:24.378682 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:24.378647    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-82njr_09c8e863-111d-467c-9c07-d0d6758b2e20/kube-multus/0.log"
Aug 04 18:56:24.845252 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:24.845226    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hmzpt_2959820d-b173-49ec-875d-9dfa00f47f50/kube-multus-additional-cni-plugins/0.log"
Aug 04 18:56:24.891349 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:24.891321    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hmzpt_2959820d-b173-49ec-875d-9dfa00f47f50/egress-router-binary-copy/0.log"
Aug 04 18:56:24.941323 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:24.941297    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hmzpt_2959820d-b173-49ec-875d-9dfa00f47f50/cni-plugins/0.log"
Aug 04 18:56:24.977240 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:24.977215    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hmzpt_2959820d-b173-49ec-875d-9dfa00f47f50/bond-cni-plugin/0.log"
Aug 04 18:56:25.006972 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:25.006948    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hmzpt_2959820d-b173-49ec-875d-9dfa00f47f50/routeoverride-cni/0.log"
Aug 04 18:56:25.037308 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:25.037283    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hmzpt_2959820d-b173-49ec-875d-9dfa00f47f50/whereabouts-cni-bincopy/0.log"
Aug 04 18:56:25.070647 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:25.070616    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hmzpt_2959820d-b173-49ec-875d-9dfa00f47f50/whereabouts-cni/0.log"
Aug 04 18:56:25.176368 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:25.176256    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-647mg_47e059c0-7685-4231-86b6-5dbea4f4d5d5/network-metrics-daemon/0.log"
Aug 04 18:56:25.206600 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:25.206569    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-647mg_47e059c0-7685-4231-86b6-5dbea4f4d5d5/kube-rbac-proxy/0.log"
Aug 04 18:56:26.772976 ip-10-0-132-242 kubenswrapper[2566]: I0804 18:56:26.772947    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfzm2_e40e6de3-aae2-44dc-aa01-ed1de3d1ce5e/ovn-controller/0.log"